Update fsssignaldiscovered-v1.0.json

Added SpawningPower property.

.editorconfig

@@ -0,0 +1,21 @@
+# This is the project top-level .editorconfig
+root = true
+
+# Defaults for all file types
+[*]
+# 4-space indents, no TABs
+indent_style = space
+tab_width = 4
+indent_size = tab
+
+# Hard-wrap at 120 columns
+max_line_length = 119
+
+# Unix EOL, single \n
+end_of_line = lf
+
+# UTF-8 is the only sensible option, no BOM
+charset = utf-8
+
+# All files should have a final newline
+insert_final_newline = true

README.md

@@ -1,7 +1,7 @@
 # EDDN - Elite Dangerous Data Network
 
 ## About EDDN
-Elite Dangerous Data Network is a tool that facilitates the players of the game
+Elite Dangerous Data Network is a tool that facilitates players of the game
 [Elite Dangerous](https://www.elitedangerous.com/), including its 
 expansions, sharing data about the game galaxy with others.
 By pooling data in a common format, tools and analyses can be produced that add
@@ -15,58 +15,154 @@ archive or "current state" available to anyone.  What it provides is a
 stream of live data to any interested parties.  Some of those then make 
 aggregated data available for general use.
 
+If you want an archive of messages, or a dump of the current known state of
+the game galaxy, check ["Archives and data dumps"](#archives-and-data-dumps)
+below.
+
+---
 ---
 
 ## Using EDDN
 ### Game players
-If you are a player of the game and only want to help out by sharing the 
-data available to you over EDDN then please consult the
-[EDCD Cmdr's Guide](https://edcd.github.io/cmdrs-guide.html).  For the most 
-part if you want to share data then you will need to be playing the game on a 
-PC, but there are some tools that utilise an API provided by the game 
+It might be useful to consult the [EDCD Cmdr's Guide](https://edcd.github.io/cmdrs-guide.html)
+for a general overview of how players can contribute and use game data.
+
+---
+
+#### Contributing data
+There are a variety of tools available to players in order for them to help 
+out by contributing data, and to then utilise that data to enhance their 
+gameplay experience.
+
+For the most part any player who wishes to share data
+will need to be playing the game on a PC, as that gives direct access to 
+"Journal" files written by the game client.  These are the best source of 
+game data.
+There are, however, some tools that utilise an API provided by the game
 developer that can supply some data if you are playing on a console.
 
+So, on PC, look into installing one of the following tools:
+
+- [E:D Market Connector](https://github.com/EDCD/EDMarketConnector/wiki)
+- [EDDI](https://github.com/EDCD/EDDI)
+- [EDDiscovery](https://github.com/EDDiscovery/EDDiscovery)
+- [Elite Log Agent](https://github.com/DarkWanderer/Elite-Log-Agent)
+
+This list is not exhaustive, or intended to particular endorse any of these 
+projects over another, listed here or not.
+
+If you are playing on console some options are:
+
+- The 'console updater', available in a user's 'dashboard' on
+  [EDSM](https://www.edsm.net).
+- [Journal Limpet](https://journal-limpet.com/).
+
+---
+
+#### Utilising data
 If you're looking for tools that utilise EDDN data to enhance your experience
 then you're probably looking for one of the sites listed below.  NB: These are
 listed in name-alphabetical order and no particular ranking or endorsement is
 intended.
 
-- [EDDB](https://eddb.io/) - a website which tries to act as a database of all
- the data available in the game. In general EDDB tries to help finding 
- stuff which players are looking for.
 - [EDSM](https://www.edsm.net/) - originally focused on being a 'Star Map', 
   but has since expanded its functionality.  Of particular interest to 
   in-game explorers.
-- [Inara](https://inara.cz/) - a popular alternative to EDDB, with a lot of 
-  its own unique functionality.
+- [Inara](https://inara.cz/) - a popular alternative to the now defunct EDDB, 
+  with a lot of its own unique functionality.
 - [Spansh](https://www.spansh.co.uk/plotter) - originally this had one tool,
   a 'Neutron Star' route plotter, but has since expanded into offering many 
   other route plotting tools and general data searching.
 
-There are many other third-party tools for Elite Dangerous listed on
-[Elite: Dangerous Codex](https://edcodex.info/), some of which will 
-interact with EDDN.  Check the [EDDN tag](https://edcodex.info/?m=tools&cat=9).
+##### Archives and data dumps
+
+Alternatively if you want an archive of past EDDN messages, or a data dump to
+use:
+
+- [edgalaxydata](https://edgalaxydata.space/) has various data captures,
+    including 'all' (some listener downtime is inevitable) EDDN messages for
+    many years.
+
+- [spansh dumps](https://www.spansh.co.uk/dumps) are a "whole galaxy" data set,
+    of systems, bodies and stations.  The full `galaxy.json.gz` is **very**
+    large, but is currently the only source of an "all known bodies" dump.
+    Pay attention to the 'Generated' "time ago" column.
+
+- [EDSM nightly dumps](https://www.edsm.net/en/nightly-dumps) represent a
+    snapshot of the data EDSM uses.  NB: there's only a "last 7 days" bodies
+    dump as the full data proved too large to dump in a timely manner.
+
+---
+
+There are many other third-party tools for Elite Dangerous, both for 
+contributing data and utilising it, listed on
+[Elite: Dangerous Codex](https://edcodex.info/).  Some of them
+interact with EDDN - check the [EDDN tag](https://edcodex.info/?m=tools&cat=9).
+
+---
 
 ### Developers
 If you are a developer of a third-party tool that could be enhanced by 
 uploading data to EDDN then please consult
-[the live branch documentation](https://github.com/EDCD/EDDN/blob/live/schemas/README-EDDN-schemas.md)
+[the live branch Developers' documentation](https://github.com/EDCD/EDDN/blob/live/docs/Developers.md)
 .
 **DO NOT** assume that any code or documentation in the `master` (or 
 any other) branch on GitHub truly reflects the current live service!
 
-### Misc
-There is also a [wiki page](https://github.com/EDSM-NET/EDDN/wiki), but its 
-contents are currently being migrated into the source code tree (so that 
-they always match the in-use code).
+Anyone planning to send data too EDDN **MUST** comply with all the advice in
+that document, and the individual schema README files as applicable.  It's
+also the best resource for those listening to the EDDN data stream.
 
+#### EDDN endpoints
+
+There are up to three separate services which might be running.
+
+| Service | Upload |           Listeners           | Notes                                                                                                                                                                                 |
+| ------: | :-----: |:-----------------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Live | `https://eddn.edcd.io:4430/upload/` |   `tcp://eddn.edcd.io:9500/`    | The actual live service, which should always be running. It is automatically restarted every Thursday at 07:06:00 UTC                                                                 |
+| Beta | `https://beta.eddn.edcd.io:4431/upload/` | `tcp://beta.eddn.edcd.io:9510/` | The beta service, which should be running the current state of the `beta` branch.  Usually only active when either new schemas or forthcoming code changes are being actively tested. |
+| Dev | `https://dev.eddn.edcd.io:4432/upload/` | `tcp://dev.eddn.edcd.io:9520/`  | The dev service, which could be running any public branch of the code *or* a private branch.                                                                                          | 
+
+In general the Beta and Dev services will only be running so as to aid the core
+development team in testing changes.  Don't expect them to be generally
+available.
+
+You **MUST** use the correct hostname in the Upload URLs as these are
+TLS/HTTPS connections terminated on a Reverse Proxy.
+
+The Listener URLs are ZeroMQ endpoints, no TLS.  But whilst this means you
+don't strictly need to use the correct hostname there is no guarantee that the
+beta and dev hostnames won't be pointing at, or hosted on, a different IP.
+
+If you need to test some of your own changes then please read
+[Running this software](docs/Running-this-software.md) for how to instantiate
+your own test service.  It is hoped that in the future the code will allow for
+easily running in a "local only" mode, not requiring any reverse proxy or
+internet-valid TLS certificates.
+
+#### New Schemas
+*All* new Schema proposals **MUST** be started by opening
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose).  Please
+consult
+[docs/Contributing 'Adding a New Schema'](docs/Contributing.md#adding-a-new-schema)
+for further guidelines.
+
+
+---
+---
+
+## Misc
+
+### Service Status
 Consult [EDDN Status](https://eddn.edcd.io/) for some information about, 
 and statistics for, the live service.
 
----
-
-## Hosting of the live service
+### Hosting of the live service
 
 Hosting is currently provided by the
 [Elite: Dangerous Community Developers](https://edcd.github.io/).
 
+### Contacting the EDDN team
+
+* [EDCD Discord - #eddn channel](https://discord.gg/DdqJ8nWVGc)
+* [E:D forum thread](https://forums.frontier.co.uk/threads/elite-dangerous-data-network-eddn.585701/#post-9400060)

contrib/eddn-logs-archive

@@ -78,7 +78,7 @@ for service in gateway monitor relay ;
 do
   log "Service: ${service}"
   log "         Expiring old logs..."
-  find . -name "${service}.log.*.gz" -a -atime +${MAX_LOGFILE_AGE} -exec rm -fv {} \;
+  find . -name "${service}.log.*.gz" -a -atime +${MAX_LOGFILE_AGE} -exec rm -f {} \;
   log "         DONE"
 
   log "         Checking if current logfile needs archiving..."
@@ -87,8 +87,9 @@ do
     log "         Archiving ${service}.log ..."
     # We have no means to tell the service to close and re-open output, it's
     # to stdout/err anyway.  So we copy it.
-    COMPRESSED_NAME="${service}.log.$(date --iso-8601=seconds)"
-    cp ${service}.log "${COMPRESSED_NAME}"
+    TIMESTAMP="$(date --iso-8601=seconds)"
+    ARCHIVED_NAME="${service}.log.${TIMESTAMP}"
+    cp ${service}.log "${ARCHIVED_NAME}"
     if [ $? -ne 0 ];
     then
       echo "         FAILED copying live log file to new archive!!!" >&2
@@ -97,8 +98,15 @@ do
     fi
     # Truncate the live file.
     :> ${service}.log
+
+    if [ "${service}" == "gateway" ];
+    then
+      # Produce a report of interesting errors
+      ${HOME}/.local/bin/eddn-report-log-errors "${ARCHIVED_NAME}" > "${HOME}/reports/eddn-errors/by-log-rotation/eddn-errors-${TIMESTAMP}.txt"
+    fi
+
     # Now compress the newly archived log
-    gzip -9v "${COMPRESSED_NAME}"
+    gzip -9 "${ARCHIVED_NAME}"
     log "         DONE"
   else
     log "         No"

contrib/monitor/index.html

@@ -53,7 +53,7 @@
                 </ul>
 
                 <p class="navbar-right navbar-text">
-                    <em>Provided by <a href="https://www.edsm.net/" target="_blank" class="navbar-link"><strong>EDSM</strong></a></em>
+                    <em>Provided by <a href="https://edcd.github.io/" target="_blank" class="navbar-link"><strong>EDCD</strong></a></em>
                 </p>
             </div>
 

contrib/monitor/schemas.html

@@ -41,7 +41,7 @@
                 </ul>
                 
                 <p class="navbar-right navbar-text">
-                    <em>Provided by <a href="https://www.edsm.net/" target="_blank" class="navbar-link"><strong>EDSM</strong></a></em>
+                    <em>Provided by <a href="https://edcd.github.io/" target="_blank" class="navbar-link"><strong>EDCD</strong></a></em>
                 </p>
             </div>
             
@@ -241,7 +241,7 @@
                     // Loop results
                     $.each(schemasCount, function(schema, hits){
                         // IF TEST CONTINUE
-                        if(schema.substr(schema.length - 4) == 'test')
+                        if(schema.slice(-4) == 'test')
                             return;
 
                         var slug = makeSlug(schema);

contrib/test-schema.py

@@ -7,7 +7,7 @@ from jsonschema import validate as json_validate
 if len(sys.argv) < 2:
     print(
 f"""
-Usage: {sys.argv[0]} <schema file name> <test data file name>
+Usage: {sys.argv[0]} <schema file name> [<test data file name>]
 
 Note that the entire file will be loaded by simpljson.load() and should
 only contain one JSON object.
@@ -16,12 +16,18 @@ only contain one JSON object.
     sys.exit(-1)
 
 schema_file_name = sys.argv[1]
-test_file_name = sys.argv[2]
+test_file_name = None
+if len(sys.argv) == 3:
+    test_file_name = sys.argv[2]
 
-with open(test_file_name, 'r') as test_file:
-    test_event = simplejson.load(test_file)
+with open(schema_file_name, 'r') as schema_file:
+    schema = simplejson.load(schema_file)
+    print('Schema file loaded OK...')
 
-    with open(schema_file_name, 'r') as schema_file:
-        schema = simplejson.load(schema_file)
+    if test_file_name is not None:
+        with open(test_file_name, 'r') as test_file:
+            test_event = simplejson.load(test_file)
 
-    json_validate(test_event, schema, format_checker=FormatChecker())
+        json_validate(test_event, schema, format_checker=FormatChecker())
+
+        print('Input file validated against schema OK.')

docs/Adding-A-New-Schema.md

@@ -1,11 +1,16 @@
 # Adding A New Schema
 
 ## Introduction
+
 As of September 2021 it was decided that all new Journal events will be
 added to their own, new, schemas.  This better facilitates defining any
 values that should be elided, or augmentations added, without twisting
 schema definitions into knots.
 
+Consult
+[Contributing.md#adding-a-new-schema](./Contributing.md#adding-a-new-schema)
+for guidelines on the specifics of proposing and designing a new Scema.
+
 In the future we will likely migrate all of the events currently
 supported in the journal schema into their own schemas, and later still
 deprecate the journal schema.

docs/Contributing.md

@@ -0,0 +1,356 @@
+# Contributing to the EDDN Project
+
+## Introduction
+
+This document is intended to solidly and usefully define necessary information
+pertaining to either improving the EDDN software, or add a new Schema to the
+supported set.
+
+---
+
+## File formatting and editor configuration
+
+The project contains an `.editorconfig` file at its root.  Please either ensure
+your editor is taking note of those settings, or cross-check its contents
+with the
+[editorconfig documentation](https://github.com/editorconfig/editorconfig/wiki/EditorConfig-Properties)
+, and ensure your editor/IDE's settings match.
+
+---
+
+## Branches and other project miscellanea
+
+This project utilises a number of Git branches:
+
+- `live` - The Live service should, outside of brief periods during deployment,
+   always be running using the software and supporting files from this branch.
+   This is where you should be looking if checking current behaviour of the
+   service.
+
+- `master` - This is in effect the staging area for any changes considered
+   ready for deployment to the Live service.
+
+- `beta` - Changes under consideration for Live deployment that are currently
+   undergoing further testing.  If the Beta service is running it should be
+   using this branch, but there might be exceptions.
+   There MAY be changes in `develop` *and* `master` that were not merged here.
+
+- `develop` - Usually any Pull Requests will be merged here first.  This is the
+   default branch against which any new work should be undertaken.  Urgent
+   bug fixes should be the only exception to this, and even then it would only
+   happen if `develop` already contains non-Live changes that are not yet
+   considered ready for deployment.
+
+You might also see 'work in progress' branches with a `fix/` or `enhancement/`
+prefix.
+
+---
+---
+
+## Code Changes
+
+All code changes should start with
+[an open Issue on GitHub](https://github.com/EDCD/EDDN/issues?q=is%3Aissue+is%3Aopen).
+If no pertinent issue already exists then please create one.
+
+All Pull Requests should be made against the `develop` branch unless you are
+directed to do otherwise.  A Pull Request that is opened without prior
+discussion in a relevant Issue is liable to be closed without further
+consideration, but exceptions may be made for 'obvious' changes.
+
+### Testing
+
+As of 2022-01-28 the project still does not contain any automated tests,
+neither unit or functional.  But you should make every effort to test any
+changes, including new Schemas, or changes to existing ones, before opening
+a Pull Request for those changes.
+
+`scripts/testing/` exists and might contain some scripts and supporting files
+that will be useful in this.
+
+### Bug Fixes
+
+An urgent Bug Fix may be fast-tracked through to the `master` branch, but will
+by default go through the `develop` branch.
+
+Where changes pertain to fixing a bug they should be in a branch named as per
+the convention `fix/<issue number>/<brief descrption>`, e.g.
+`fix/123/avoid-decompress-crash`.
+
+### Enhancements
+
+Any changes to existing code or supporting files that does not address a bug
+is considered an enhancement.  Examples would be:
+
+- Changes to an existing Schema to better support actual game data and
+   potential uses by Listeners.  If you're not sure whether the change is an
+   Enhancement or Fix, use your best assessment, we won't bite your head off.
+- Adding a wholly new Schema.
+- Improving the Monitor web page, be that adding extra output or a new way to
+  view or manipulate the data presented.
+
+Where changes pertain to adding wholly new functionality, including adding a
+new schema, or improving an existing feature, then they should be in a branch
+named as per the convention `enhancement/<issue number>/<brief descrption>`
+, e.g. `enhancement/234/add-schema-somenewevent`.
+
+---
+---
+
+## Adding a New Schema
+
+If you think you have a good case for an additional EDDN Schema then there are
+several things you should consider:
+
+1. Ensure you read
+   [the general Schemas README](https://github.com/EDCD/EDDN/blob/live/schemas/README-EDDN-schemas.md)
+   to be aware of general requirements that your new Schema will need to
+   adhere to.
+
+   You might also find useful information in the other Schema-specific README
+   files.  Certainly check those if you encounter trouble documenting a new
+   Schema.
+
+2. What is the source of the data?  In almost all circumstances the only 
+   acceptable sources are the game Journal files and the Frontier CAPI service.
+   We do *NOT* accept any manually entered data being sent over EDDN, it's too
+   prone to error.
+
+   - Do **NOT** blindly trust that the Frontier-provided Journal documentation
+      is correct with respect to the current game version.  Gather actual
+      examples of Journal output under as varied circumstances as are relevant,
+      and base your new Schema on what you learn.
+   
+   - Remember that there might be differences between data from a player using
+      a Horizons version of the game versus an Odyssey version.  This is why
+      all Schemas should mandate augmentation with `horizons` and `odyssey`
+      flags, but there might be other considerations when defining a Schema.
+
+3. Is the new Schema going to be practically useful ?
+   1. What use cases are there for the new data?
+   2. Given that nowhere near all players will be running an EDDN
+      sender, and even where they do we might still miss some relevant data,
+      is this data still useful ?
+   
+      e.g. the owner of a Fleet Carrier sending data about their buy and sell
+      orders is useful, but if they then don't log in again for a while
+      there'll be no update to the FC state.  Likewise for an FC jumping
+      between systems.
+   
+      At the very least you should consider, and document, caveats about the
+      data for the benefit of Listeners.
+   3. Who benefits ?  If the only interested Listener would be a very niche
+      project, with no benefit to others, then perhaps you should instead
+      consider e.g. an EDMarket Connector plugin that sends to your own
+      server ?
+   4. What's the likely volume of messages compared to existing Schemas?  If
+      there would often be many messages in a short space of time consider
+      requiring senders to batch them.  2022-01-28: There's no live example of
+      this yet, but see
+      [discussion of adding support for FSSSignalDiscovered](https://github.com/EDCD/EDDN/issues/152)
+      .
+   5. What's the likely size range of the new messages?  The Gateway has a
+      limit on the size of the *body* of `/upload/` requests.  Check
+      [live branch src/eddn/Gateway.py](https://github.com/EDCD/EDDN/blob/live/src/eddn/Gateway.py)
+      `bottle.BaseRequest.MEMFILE_MAX = ...` for the current limit.
+
+4. For CAPI-sourced data you need to keep in mind possible synchronization
+   issues between it and any necessary data augmentations from Journal data.
+   This might mean needing to make such augmentations optional.
+
+5. For Journal-sourced data if the source is an event not yet allowed by any
+   existing Schema then you MUST define a wholly new Schema for the data.  This
+   allows you to fully specify both required and forbidden information.
+
+   The Journal events that are handled in the generic `journal` Schema are only
+   there for historical reasons and due to the difficulties in ensuring all
+   listeners and senders migrate to separate Schemas in a synchronized manner.
+
+6. You **MUST**
+   [open an issue on GitHub](https://github.com/EDCD/EDDN/issues/new)
+   in order to propose the new Schema.  If a consensus appears to have been
+   reached in comments therein then start work on a Pull Request.
+
+7. Consult
+   [the template for new Journal-based schemas](../schemas/TEMPLATES/journalevent-v1.0.json)
+   and [its README template](../schemas/TEMPLATES/journalevent-README.md) for
+   an outline and some further guidance.
+
+8. There must be at least one working Sender implementation before the Pull
+   Request for a new Schema will be merged into the Live service.  Experience
+   has demonstrated that there are often caveats and gotchas discovered during
+   the development of a Sender for a new Schema.
+
+   Often this will end up being a Pull Request against either
+   [Elite Dangerous Market Connector](https://github.com/EDCD/EDMarketConnector) 's
+   EDDN plugin, or [ED Discovery](https://github.com/EDDiscovery/EDDiscovery).
+
+The Schema files are placed in the `schemas/` directory, located in the root
+of the project structure.  See [Schema file requirements](#schema-file-requirements)
+for more information.
+
+---
+
+### Always start a new Schema at version 1
+
+The first time a new Schema goes live it should be as version 1.
+ - What should policy be on incrementing the version ?  I'm not confident
+   anything other than an integer is supported - Ath
+
+Any breaking changes **MUST** increment the version number.  Use a git file
+rename to update the name of the file.  Examples of such breaking changes
+include:
+
+- If you add a new required property.  Senders will need to update.
+- If you remove a required property *and making it optional doesn't make
+  sense*.  Senders will need to update.  Listeners will need to cope with the
+  data no longer being present.
+- If you change a property from optional to required or disallowed.  Senders
+  will need to update.  Listeners can no longer expect it, if disallowed.
+
+---
+
+### Necessary file edits
+
+1. Obviously you need to create the new file in the `schemas/` directory.
+  This should be named as per the data source, i.e. Journal `event` value, and
+  include the Schema version, and `.json` extension.  You **MUST** fold the
+  the `event` value to lower case for this.
+  An example is `fssdiscoveryscan-v1.0.json` for adding support for the Journal
+  `FSSDiscoveryScan` event.
+
+2. You **MUST** also create the README file for the new Schema.  This is also
+  placed in the `schemas/` directory.  The name should match that of the
+  Schema file itself, without the version, and with a `.md` extention instead
+  of `.json`.
+  An example is `fssdiscoveryscan-README.md` documenting the
+  `fssdiscoveryscan-v1.0.json` Schema file.
+
+3. You will need to add two lines to `src/eddn/conf/Settings.py` in order to
+  have the Gateway recognise the new Schema.  You are adding to the
+  end of the `GATEWAY_JSON_SCHEMAS` dictionary.  Both the live Schema *and*
+  the `/test` version **MUST** be added.
+  For `fssdiscoveryscan-v1.0.json` you would add:
+    ```python
+   
+        "https://eddn.edcd.io/schemas/fssdiscoveryscan/1"                    : "schemas/fssdiscoveryscan-v1.0.json",
+        "https://eddn.edcd.io/schemas/fssdiscoveryscan/1/test"               : "schemas/fssdiscoveryscan-v1.0.json",
+    ```
+   Please ensure you use the current hostname as per the entries for already
+   existing Schemas.  Keep the trailing comma on the final entry, Python
+   allows it, and it will reduce the diff on adding any further Schemas.
+
+4. You MUST add a file containing an example **valid** full EDDN message in the
+   `scripts/testing/gateway-responses/` directory.
+   1. Name the file as per the Schema name.  In the case of only adding a
+      single valid message you can use e.g. `newschema.json`.
+       1. If adding variants of valid messages then please annotate the
+          filename appropriately, e.g. `newschema-valid-inspace.json` and
+          `newschema-valid-onbody.json`.
+       2. If adding variants that are *invalid* in some manner them name
+          the files as e.g. `newschema-invalid-no-starpos.json`.
+   2. The file MUST contain the full plain-text of an EDDN upload, as would be
+      sent in the body of a request to the `/upload/` endpoint.  Test it with
+      the `scripts/testing/gateway-response/test-sender.py` script.
+   3. Please have the `$schemaRef` key:value first, followed by the `header`
+      dictionary, and only then the `message` dictionary.
+   4. Base the `message` part of this on actual source data, e.g. a line
+      from a relevant Journal file.
+   5. Ensure the message `timestamp` value is sane, i.e. from a time period
+      in which the game was providing this data.
+   6. Ensure any data added as an augmentation is correct, i.e.
+      the proper co-ordinates of the named StarSystem in StarPos.
+
+   This will aid in confirming that the new schema actually works for a valid
+   message.  You MAY add additional examples that are invalid in various ways
+   that the schema will detect, but this is not required.
+
+#### Schema file requirements
+
+
+1. The file **MUST** be valid [JSON](https://www.json.org/json-en.html),
+   without any special extensions (so no comments).  Remember that JSON does
+   **not** allow for a trailing comma on the last entry of an array or
+   dictionary.
+2. The file **MUST** comply with the relevant
+   [JSON Schema](https://json-schema.org/) definition.
+
+   As of 2022-01-28 we still use 'draft 04' of this specification.
+   We are looking into updating to the latest in
+   [#139 - Update to latest JSON schema version(s) ](https://github.com/EDCD/EDDN/issues/139).
+3. The file **MUST** load using Python's `simplejson` module, as this
+   is what the Gateway code uses.  The script `contrib/test-schema.py` will
+   check both this and that the validation code doesn't choke on it.
+4. All new Schemas **MUST** comply with all requirements outlined in the
+   [general Schemas documentation](../schemas/README-EDDN-schemas.md).
+   If you have a good reason why your new Schema can't and shouldn't comply
+   with these requirements, then consensus will need to be achieved on changing
+   those requirements and/or allowing the exception.
+5. If the data source is a game Journal event then you **MUST** include the
+   `event` key and its value as in the source data.  This might seem redundant
+   when we mandate a separate Schema for any newly handled Journal event, but
+   it does no harm and might make data handling for Listeners easier, i.e.
+   they can just pass all "oh, that's from Journal data" messages through the
+   same initial handling.
+6. All Schema files MUST be accompanied by a MarkDown formatted
+   [README file](#schema-readme-requirements).
+
+#### Data that should be allowed in a new Schema
+
+The default is to define the new Schema such that all possible data in the
+source Journal event is allowed.
+
+However, you **MUST** consider which source data constitutes information
+personal to the player, e.g. their credit balance, or the specific amount they
+paid for something after the applicable discounts and mandate its removal in
+the Schema.
+
+Furthermore, the Schema **MUST** mandate removal of all keys (and thus their
+values as well) where the key name has the suffix `_Localised`.  These
+key:value pairs should be accompanied in the source data by a non-Localised
+version which will remain, unless otherwise elided.
+
+You should gather as many examples as possible of actual game Journal events,
+and then define the Schema such that anything (after mandated elisions as
+above) that is always in the source data is *required* in the new Schema.
+
+Anything that does not always occur in the source data, and is not defined and
+as a mandatory elision, should be defined, but remain optional.
+
+#### Schema README requirements
+
+The combination of the Schema file itself and its README **MUST** give both
+Senders and Listeners sufficient information to correctly handle the pertinent
+data.
+
+If the Schema signifies a field should be removed then you do not need
+to explicitly call it out in the README.
+
+Likewise, the Schema MUST note, with a `renamed` property, where the key name
+for some data differs from that in the Journal source data.  Because this is
+mandatory you do not then need to list such in the README.
+
+You do not need to repeat anything already specified in the general
+Schema README.  Referring to it via MarkDown linking is helpful.
+
+1. Any augmentations to a message must be clearly explained.
+   1. **DO** outline where the additional data comes from.  e.g. `StarPos`
+      added to many events should come from a prior `Location`, `FSDJump` or
+      `CarrierJump` Journal event.
+
+2. The reason(s) why any property is optional must be clearly explained.
+   Perhaps it's not always present in the source data.
+
+3. The reason(s) why any data in the source is not in the message, i.e. because
+   it's personal to the player, or maybe it's just not useful (always the same
+   in every instance of the source data).
+
+4. If your Schema only works whilst not complying with any main Schema 
+   requirements, and this has been approved, then you need to explicitly
+   document which requirement(s) are waived and why.
+
+5. If you use another Schema's README as the basis for yours then you MUST
+   remove any text that isn't relevant to your Schema.
+
+---
+---

docs/Developers.md

@@ -0,0 +1,610 @@
+## Introduction
+
+EDDN is a
+[zeromq](https://zeromq.org/) service which allows players of the game
+[Elite Dangerous](https://www.elitedangerous.com/), published
+by [Frontier Developments](https://www.frontier.co.uk/), to upload game data so
+that interested listeners can receive a copy.
+
+EDDN accepts HTTP POST uploads in
+[a defined format](#sending-data)
+representing this game data and then passes it on to any interested listeners.
+
+---
+
+## Sources
+
+There are two sources of game data, both provided by the publisher of the game,
+Frontier Developments.  They are both explicitly approved for use by
+third-party software.
+
+### Journal Files
+
+On the PC version of the game, "Journal files" are written during any game
+session. These are in newline-delimited JSON format, with each line
+representing a single JSON object. Frontier Developments publishes
+documentation for the various events in their
+[Player Tools & API Discussions](https://forums.frontier.co.uk/forums/elite-api-and-tools/)
+forum.
+
+In general the documentation is made available in a file named something like:
+
+    Journal_Manual-v<version>
+
+as both a MicroSoft word `.doc` file, or a `.pdf` file.  Historically the
+use of `_` versus `-` in those filenames has varied.
+
+Consult the latest of these for documentation on individual events.  
+However, be aware that sometimes the documentation is in error, possibly due to
+not having been updated after a game client change.
+
+### Companion API (CAPI) data
+
+Frontier Developments provides an API to retrieve certain game data, even
+without the game running.  Historically this was for use by its short-lived
+iOS "Companion" app, and was only intended to be used by that app. There was no
+public documentation, or even admission of its existence.
+
+Eventually, after some enterprising players had snooped the connections and
+figured out the login method and endpoints, Frontier Developments
+[allowed general use of this](https://forums.frontier.co.uk/threads/open-letter-to-frontier-developments.218658/page-19#post-3371472)
+.
+
+Originally the API authentication required being supplied with the email and
+password as used to login to the game (but at least this was over HTTPS).
+
+In late 2018 Frontier switched the authentication to using an oAuth2 flow,
+meaning players no longer need to supply their email and password to
+third-party sites and clients.
+
+As of October 2021 there has still never been any official documentation about
+the available endpoints and how they work. There is some
+[third-party documentation](https://github.com/Athanasius/fd-api/blob/main/docs/README.md)
+by Athanasius.
+
+It is *not* recommended to use CAPI data as the source as it's fraught with
+additional issues.  EDMarketConnector does so in order to facilitate
+obtaining station data without the player needing to, e.g. open the commodities
+screen.
+
+#### Detecting CAPI data lag
+
+When using the Companion API please be aware that the server that supplies this
+data sometimes lags behind the game - usually by a few seconds, sometimes by
+minutes. You MUST check in the data from the CAPI that the Cmdr is
+docked, and that the station and system names match those
+reported from the Journal before using the data for the commodity, outfitting
+and shipyard Schemas:
+
+1. Retrieve the commander data from the `/profile` CAPI endpoint.
+2. Check that `commander['docked']` is true.  If not, abort.
+3. Retrieve the data from the `/market` and `/shipyard` CAPI endpoints.
+4. Compare the system and station name from the CAPI market data,
+   `["lastStarport"]["name"]` and `["lastSystem"]["name"]`,
+   to that from the last `Docked` or `Location` journal event.  If either does
+   not match then you MUST **abort**.  This likely indicates that the CAPI
+   data is lagging behind the game client state and thus should not be used.
+
+---
+
+## Uploading messages
+
+### Send only live data to the live Schemas
+You MUST **NOT** send information from any non-live (e.g. alpha or beta)
+version of the game to the main Schemas on this URL.
+
+You MAY send such to this URL so long as you append `/test` to the `$schemaRef`
+value, e.g.
+
+    "$schemaRef": "https://eddn.edcd.io/schemas/shipyard/2/test",
+
+You MUST also utilise these test forms of the Schemas whenever you are
+testing your EDDN-handling code, be that new code or changes to existing code.
+
+As well as the Live service there are also `beta` and `dev`
+[endpoints](../README.md#eddn-endpoints) which might be available from time
+to time as necessary, e.g. for testing new Schemas or changes to existing
+ones.  Ask on the `#eddn` channel of the
+[EDCD Discord](https://edcd.github.io/) (check at the bottom for the invite
+link).
+
+Alternatively you could attempt
+[running your own test instance of EDDN](./Running-this-software.md).
+
+### Sending data
+Messages sent to EDDN **MUST**:
+
+- Use the URL: `https://eddn.edcd.io:4430/upload/`
+   - Note the non-standard port `4430`.
+   - Yes, the trailing `/` is required.
+   - Note the use of TLS-encrypted HTTPS.  A plain HTTP request will elicit a `400 Bad
+  Request` response.
+- Use the HTTP 1.1 protocol.  HTTP/2 is not supported at this time.
+- Use a **POST** request, with the body containing the EDDN message.  No
+  query parameters in the URL are supported or necessary.
+
+The body of an EDDN message is a JSON object in UTF-8 encoding.  If you do not
+compress this body then you MUST set a `Content-Type` header of
+`applicaton/json`.
+
+You *MAY* use gzip compression on the body of the message, but it is not
+required.  If you do compress the body then you **MUST* send a
+`Content-Encoding` header of `gzip`.
+
+**Due to issues when messages are compressed, form-encoded data is NO LONGER
+SUPPORTED as of 2022-06-16.**
+
+You should be prepared to handle all scenarios where sending of a message
+fails:
+
+1. Connection refused.
+2. Connection timed out.
+3. Other possible responses as documented in
+   [Server responses](#server-responses).
+
+Carefully consider whether you should queue a 'failed' message for later
+retry.  In particular, you should ensure that one 'bad' message does not
+block other messages from being successfully sent.
+
+You **MUST** wait some reasonable time (minimum 1 minute) before retrying
+any failed message.
+
+You **MUST NOT** retry any message that received a HTTP `400` or `426` code.
+An exception can be made if, **and only if**, *you have manually verified that
+you have fixed the issues with it (i.e. updated the Schema/version to a
+currently supported one and adjusted the data to fit that Schema/version).*
+
+You **MAY** retry a message that initially received a `413` response (in
+the hopes that the EDDN service admins decided to increase the maximum
+allowed request size), but should not do so too quickly or in perpetuity.
+
+In general:
+
+- No data is better than bad data.
+- *Delayed* good data is better than degrading the EDDN service for others.
+
+### Format of uploaded messages
+Each message is a JSON object in UTF-8 encoding with contents that comply with
+the relevant Schema.  There is an outline in the general
+[EDDN Message README](../schemas/README-EDDN-schemas.md#mandatory-schema-file-contents).
+
+For example, a shipyard message, version 2, might look like:
+
+```JSON
+{
+  "$schemaRef": "https://eddn.edcd.io/schemas/shipyard/2",
+  "header": {
+    "uploaderID": "Bill",
+    "gameversion": "4.0.0.1451", 
+    "gamebuild": "r286916/r0 ",
+    "softwareName": "My excellent app",
+    "softwareVersion": "0.0.1"
+  },
+  "message": {
+    "systemName": "Munfayl",
+    "stationName": "Samson",
+    "marketId": 128023552,
+    "horizons": true,
+    "odyssey": true,  
+    "timestamp": "2022-09-27T06:39:43Z",
+    "ships": [
+      "anaconda",
+      "dolphin",
+      "eagle",
+      "ferdelance",
+      "hauler",
+      "krait_light",
+      "krait_mkii",
+      "mamba",
+      "python",
+      "sidewinder"
+    ]
+  }
+}
+```
+
+---
+
+### Contents of `header`
+You **MUST** send the `header` component of the message.
+
+#### uploaderID
+The EDDN Relay will obfuscate the `uploaderID` value to prevent long-term
+tracking of individual players.  Please **DO** send a sensible
+`uploaderID` value, preferably simply the relevant in-game Commander name.
+
+#### softwareName
+You **MUST** set a unique, and self-consistent, value of `softwareName` so
+that you can be easily identified should any issues be found with the messages
+you send.
+
+#### softwareVersion
+You **MUST** set a pertinent value for `softwareVersion`.  We would recommend
+using [Semantic Versionining](https://semver.org/#semantic-versioning-specification-semver)
+in your project.
+
+Listeners MAY make decisions on whether to accept data, or to treat it
+differently, based on this.  As such you **MUST** increment your version
+number if you make any changes to the content of messages your software sends
+to EDDN.
+
+#### `gameversions` and `gamebuild`
+To ensure that Listeners can make decisions on how to handle data based on
+the client and feature set it came from there are two mandatory fields in
+the headers of EDDN messages.  NB: Initially the *schemas* do not actually
+make these mandatory, **but all Senders MUST make every effort to include
+them ASAP**.
+
+Where present in the data source the `gameversion` value **MUST** come from
+the field of that name in the data source, i.e. from either `Fileheader` or
+`LoadGame` as outlined below.  
+
+1. If you are using Journal files directly then you **MUST** use the value
+  from the`Fileheader` event.  Values from `LoadGame` are also acceptable, but
+  there are some events before that which might be relevant to EDDN messages,
+  especially as the ordering of 'startup'/login events can change with game
+  patches.
+2. If you are using the CAPI `/journal` endpoint to retrieve and process
+  Journal events then:
+   1. You will not have `Fileheader` available.
+   2. If the field is present in the `LoadGame` event, as in 4.0 clients,
+     use its value.
+   3. If `LoadGame` does not have the field, as with 3.8 Horizons
+     clients (up to at least `3.8.0.1400`), you **SHOULD** set the value to
+     `"CAPI-Legacy-journal"`. If, for reasons of code architecture, you are
+     unable to determine that data was CAPI-sourced then you MAY set it to
+     `""` instead, **but this is only a fallback and you should endeavour to
+     send a more useful value**.
+3. If you are sourcing data from other CAPI endpoints, i.e. for commodity,
+  shipyard, outfitting or fcmaterials_capi messages, then you **SHOULD** set
+  the values appropriately as per the CAPI host and endpoint the data came
+  from.  You **MUST NOT** use the value from any Journal you are also reading 
+  for the user.  The general form is `"CAPI-(Live|Legacy)-<endpoint>"`.
+   1. If it's a commodity message from the Live CAPI host, then use
+     `"CAPI-Live-market"`.  If it was from the Legacy CAPI host then use
+     `"CAPI-Legacy-market"`.
+   2. If it's a shipyard message, then use `"CAPI-Live-shipyard"` or
+     `"CAPI-Legacy-shipyard"`.
+   3. If it's an oufitting message, then also use `"CAPI-Live-shipyard"` or
+     `"CAPI-Legacy-shipyard"`.
+   4. If it's an fcmaterials_capi message, then use `"CAPI-Live-market`" or
+     `"CAPI-Legacy-market"`, as the data comes from that endpoint.
+
+   Again, if your code architecture doesn't allow for signalling that the data
+  source was CAPI, then you MAY set it to `""` instead, **but this is only a
+  fallback and you should endeavour to send a more useful value**.
+
+For `gamebuild` you **MUST** use the value of the `build` field in the data
+source, following the same logic as for `gameversion` above, else send as `""`.
+An alternative to the latter is to mirror the `gameversion` value, **but only
+where that has not come from a Journal value**, i.e. `CAPI-...` strings.
+Do **NOT** strip any leading or trailing white space, pass it through as-is.
+
+For emphasis, *if you cannot set a data-source value, or an appropriate
+`"CAPI-..."` value then you **MUST** still send the field with an empty string
+value.*
+
+Examples of valid values:
+
+| Data Source |     Data Type | Game Galaxy | gameversion         | gamebuild           |
+|------------:|--------------:|:-----------:|:----------------------|:----------------------|
+|     Journal |       Journal |    Live     | `"4.0.0.1475"`          | `"r289563/r0 "`         |
+|     Journal |       Journal |   Legacy    | `"3.8.0.1400"`          | `"r289583/r0 "`         |
+| CAPI |    `/journal` |  Live[^1]   | `"4.0.0.1475"`          | `"r289563/r0 "`         |
+| CAPI |    `/journal` | Legacy[^2]  | `"CAPI-Legacy-journal"` | `""`                    |
+| CAPI |    `/journal` | Legacy[^2]  | `"CAPI-Legacy-journal"` | `"CAPI-Legacy-journal"` |
+| CAPI | `/market`[^3] |    Live     | `"CAPI-Live-market"`    | `""`                    |
+| CAPI | `/market`[^3] |    Live     | `"CAPI-Live-market"`    | `"CAPI-Live-market"`    |
+| CAPI | `/market`[^3] |   Legacy    | `"CAPI-Legacy-market"`    | `""`                    |
+| CAPI | `/market`[^3] |   Legacy    | `"CAPI-Legacy-market"`    | `"CAPI-Legacy-market"`    |
+[^1]: Live is a 4.0 client and has `gameversion` in `LoadGame` which is present
+  in CAPI-sourced journals.
+
+[^2]: Legacy is a 3.8 client and last tested still doesn't have `gameversion`
+  in `LoadGame`, and CAPI-sourced journals don't have `Fileheader`.
+
+[^3]: And similarly for other CAPI endpoints, e.g. `shipyard`.
+
+
+---
+
+### Contents of `message`
+Every message MUST comply with the Schema its `$schemaRef` value cites.  Each
+Schema file should have a matching `<schema>-README.md` file in the
+[project root schemas/ directory](../schemas/)
+.  Always consult this so that you're aware of any Schema-specific requirements.
+
+The Schema file, `<schema>-v<version>.json`, is considered the authority on
+the format of messages for that Schema.  If anything in the accompanying
+documentation is in disagreement with this then please
+[open an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+so that we can investigate and correct, or clarify, as necessary.
+
+Apart from short time windows during deployment of a new version the live
+EDDN service should always be using
+[the Schemas as present in the live branch](https://github.com/EDCD/EDDN/tree/live/schemas).
+So, be sure you're checking the live versions and not, e.g. those in the
+`master` or other branches.
+
+Please consult the
+[general README for Schemas](../schemas/README-EDDN-schemas.md#general-eddn-message-outline)
+for more detailed information about a message's content.
+
+EDDN is intended to transport generic data not specific to any particular Cmdr
+and to reflect only the data that every player would see in-game in station
+services or the local map. To that end:
+1. Uploading applications MUST ensure that messages do not contain any
+   Cmdr-specific data (other than "uploaderID", the "horizons" flag, and
+   the "odyssey" flag).
+2. Uploading applications MUST remove any data where the name of the
+   relevant key has a `_Localised` suffix.
+
+The individual Schemas will instruct you on various elisions (removals) to
+be made to comply with this.
+
+Because the first versions of some Schemas were defined when only the CAPI
+data was available, before Journal files existed, many of the key names chosen
+in the Schemas are based on the equivalent in CAPI data, not Journal events.
+This means you MUST rename many of the keys from Journal events to match the
+Schemas.  Consult the relevant Schema for details.
+
+Some of these requirements are also enforced by the Schemas, and some things
+the Schemas enforce might not be explicitly called out in documentation.  So,
+**do** check what you're sending against the relevant Schema(s) when making any
+changes to your code.
+
+It is also advisable to Watch this repository
+[on GitHub](https://github.io/EDCD/EDDN/)
+so that you are aware of any changes to Schemas.
+
+#### `horizons` and `odyssey` flags
+
+Where the Schema allows for them, `horizons` and `odyssey` keys **MUST** be
+added with appropriate boolean values.  `null` is not allowed in the values,
+so **if you cannot determine a value do not include that key at all**.
+
+To emphasise that, *in the case where there is no `Odyssey` boolean in the
+`LoadGame` event* **DO NOT INCLUDE IT IN THE EDDN MESSAGE**.  No, not with a
+`false` value.  **DO NOT INCLUDE IT**.
+
+The only source of these is the `LoadGame` event from journals.  It's present
+both in the PC local files and the CAPI journal data.  If you're
+composing a shipyard or outfitting message from CAPI data then it is
+possible to synthesise the `horizons` flag.  For now consult the function
+`capi_is_horizons()` in
+[EDMarketConnector:plugins/eddn.py](https://github.com/EDCD/EDMarketConnector/blob/stable/plugins/eddn.py)
+for a method to achieve this.
+
+As of 2022-09-27 the following was observed for the `LoadGame` events as
+present in CAPI-sourced Journal files (which were confirmed to match the
+PC-local files for these events):
+
+- PC Odyssey Client, game version `4.0.0.1450`:
+    ```json
+    { "timestamp":"2022-09-27T09:47:35Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true, "Odyssey":true, ...
+    ```
+- PC Horizons 4.0 Client, game version `4.0.0.1450`:
+    ```json
+    { "timestamp":"2022-09-27T11:25:45Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true, "Odyssey":false, ...
+    ```
+- PC Horizons Client, game version `3.8.0.407`, no `Odyssey` key was
+  present:
+    ```json
+    { "timestamp":"2022-09-27T11:28:53Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true, ...
+    ```
+
+- PC 'base' Client, game version `3.8.0.407`, no `Odyssey` key was
+  present:
+    ```json
+    { "timestamp":"2022-09-27T11:31:32Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":false, ...
+    ```
+
+Do not attempt to use the value(s) from a `Fileheader` event as the semantics
+are different.  With clients 3.8.0.407 and 4.0.0.1450 the following was observed:
+
+| Game Client      | Fileheader      | LoadGame                         |
+| ---------------: |-----------------|---------------------------------:|
+| Base             | "Odyssey":false |                 "Horizons":false |
+| Horizons 3.8     | "Odyssey":false |                  "Horizons":true |
+| Horizons 4.0     | "Odyssey":true  | "Horizons":true, "Odyssey":false |
+| Odyssey          | "Odyssey":true  |  "Horizons":true, "Odyssey":true |
+
+NB: The 'Base' client appears to simply be the Horizons client with any
+Horizons-only features disabled.
+
+- In the `Fileheader` event the `Odyssey` flag is indicating whether it's a
+  `4.0` game client.
+- In the `LoadGame` event the `Horizons` and `Odyssey` flags indicate if those
+  features are active, but in the `3.8` game client case you only get the
+  `Horizons` boolean.
+
+#### Other data Augmentations
+Some schemas mandate that extra data be added, beyond what is in the source
+data, to aid Listeners.
+
+This is usually related to specifying which system an event took place in, and
+usually means ensuring there is the full set of:
+
+1. `StarSystem` - the name of the system.
+2. `SystemAddress` - the game's unique numerical identifier for the system.
+3. `StarPos` - The system's co-ordinates.
+
+Whilst it can be argued that any Listener should see preceding event(s) that
+give any missing information where at least the system name or `SystemAddress`
+is already in the event data, this might not always be true.  So Senders MUST
+add this data where required.  It helps to fill out basic system information
+(name, SystemAddress and co-ordinates).
+
+However, there is a known game bug that can result in it stopping writing to
+the game journal, and some observed behaviour implies that it might then later
+resume writing to that file, but with events missing.  This means any Sender
+that blindly assumes it knows the current system/location and uses that for
+these Augmentations might send erroneous data.
+
+1. **Senders MUST cross-check available event data with prior 'location'
+  event(s) to be sure the correct extra data is being added.**
+2. **Listeners SHOULD realise that any data added as an Augmentation might be
+  in error.**
+
+For Senders, if the source data only has `SystemAddress` then you MUST check
+that it matches that from the prior `Location`, `FSDJump` or `CarrierJump`
+event before adding `StarSystem` and `StarPos` data to a message.  Drop the
+message entirely if it does not match.  Apply similar logic if it is only
+the system's name that is already present in data.  Do not blindly add
+`SystemAddress` or `StarPos`.  Likewise, do not blindly add `StarPos` if the
+other data is already in the source, without cross-checking the system name
+and `SystemAddress`.
+
+Listeners might be able to apply their own cross-check on received messages,
+and use any mismatch with respect to what they already know to make a decision
+whether to trust the augmented data.  Flagging it for manual review is probably
+wise.
+
+### Server responses
+There are three possible sources of HTTP responses when sending an upload
+to EDDN.
+
+1. The reverse proxy that initially accepts the request.
+2. The python `bottle` module that the Gateway uses to process the
+   forwarded requests.  This might object to a message before the actual
+   EDDN code gets to process it at all.
+3. The actual EDDN Gateway code.
+
+Once a message has cleared the EDDN Gateway then there is no mechanism for any
+further issue (such as a message being detected as a duplicate in the
+Monitor downstream of the Gateway) to be reported back to the sender.
+
+To state the obvious, if there are no issues with a request then an HTTP
+200 response will be received by the sender.  The body of the response
+should be the string `OK`.
+
+#### Reverse Proxy responses
+In addition to generic "you typoed the URL" and other such "you just didn't
+make a valid request" responses you might experience the following:
+
+1. `408` - `Request Timed Out` - the sender took too long to make/complete
+   its request and the reverse proxy rejected it as a result.
+2. `503` - `Service Unavailable` - the EDDN Gateway process is either not
+   running, or not responding.
+3. `400` - `Bad Request` - if you attempt to use plain HTTP, rather than
+   HTTPS.
+
+#### bottle responses
+1. `413` - `Payload Too Large` - `bottle` enforces a maximum request size
+   and the request exceeds that.  As of 2022-01-07 the limit is 1MiB,
+   which is versus the compressed size of the body, if compression is
+   used.  Thus compression *will* allow for sending approximately 10x
+   larger messages.
+   To verify the current limit check for the line that looks like:
+
+      ```
+      bottle.BaseRequest.MEMFILE_MAX = 1024 * 1024 # 1MiB, default is/was 100KiB
+      ```
+
+   in
+   [src/eddn/Gateway.py](https://github.com/EDCD/EDDN/blob/live/src/eddn/Gateway.py),
+   as added in
+   [commit  0e80c76cb564771465f61825e694227dcc3be312](https://github.com/EDCD/EDDN/commit/0e80c76cb564771465f61825e694227dcc3be312).
+
+#### EDDN Gateway responses
+For all failures the response body will contain text that begins `FAIL: `.  Currently two different HTTP status codes are utilised:
+
+1. `400` - `Bad Request` - This indicates something wrong with the request
+   body.  Possibly due to a format issue (compression, form encoding), or
+   the actual content of the EDDN message:
+    1. `FAIL: zlib.error: <detail>` - A failure to decompress a message that
+       claimed to be compressed.
+
+    2. `FAIL: Malformed Upload: <detail>` - the message appeared to be
+       form-encoded, but either the format was bad or there was no `data`
+       key.
+
+    3. `FAIL: JSON parsing: <detail>` - the
+       message couldn't be parsed as valid JSON.  e.g.
+
+   ```
+   FAIL: JSON parsing: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
+   ```
+
+    4. `FAIL: Schema Validation: <detail>` - the message failed to validate
+       against the cited Schema. e.g.
+
+   ```
+   FAIL: Schema Validation: [<ValidationError: "'StarPos' is a required property">]
+   ```
+
+   The exact detail will be very much dependent on both the Schema the
+   message cited and the contents of the message that failed to pass the
+   validation.
+
+   In particular, if the message contains a key that is tagged 'disallowed' in
+   the Schema, then the response will look like:
+
+   ```
+   FAIL: Schema Validation: "[<ValidationError: "{'type': ['array', 'boolean', 'integer', 'number', 'null', 'object', 'string']} is not allowed for 'BadVALUE'">]"
+   ```
+   This is due to the use of a JSON schema stanza that says "don't allow
+   any valid type for the value of this key" to trigger the error for such
+   disallowed keys.
+
+   Note how the **value** for the disallowed key is cited, not the key *name*
+   itself.  This is a limitation of how the `jsonschema` python module
+   reports errors, and we are
+   [hoping to improve this](https://github.com/EDCD/EDDN/issues/163).
+
+2. `426` - `Upgrade Required` - This indicates that the cited Schema, or
+   version thereof, is outdated.  The body of the response will be:
+
+   ```
+   FAIL: Oudated Schema: The schema you have used is no longer supported. Please check for an updated version of your application.
+   ```
+   The wording here is aimed at users of applications that send messages
+   over EDDN.  If you're the developer of such an application then
+   obviously you need to update your code to use a currently supported
+   Schema and version thereof.
+
+
+There shouldn't be any other variants of a 'FAIL' message.  If you find
+any then please
+[open an issue on GitHub](https://github.com/EDCD/EDDN/issues/new)
+with as much detail as possible so that we can update this documentation.
+
+## Receiving messages
+
+EDDN provides a continuous stream of information from uploaders. To use this
+data you'll need to connect to the stream using ZeroMQ (a library is probably
+available for your language of choice).
+
+The URL for the live Relay is:
+
+    tcp://eddn.edcd.io:9500
+
+Depending on the programming language and library used, you might need
+to explicitly subscribe to an empty topic, `''`, in order to receive
+anything.
+
+Unfortunately at this time we're using an old version of
+ZeroMQ which does not support server-side subscription filtering.  So by
+all means specify a more specific topic filter if it suits your needs,
+but realise that the dropping of not-matching messages will happen at
+your end, not the server.
+
+Once you've connected to that you will receive messages.  To access the
+data you will first need to zlib-decompress each message.  Then you will
+have a textual JSON object as per the Schemas.
+
+In general, check the guidance for [Uploading messages](#uploading-messages)
+for the expected format of the messages.  **Pay particular attention to any
+schema-specific Augmentations**.  Whilst Senders MUST make every effort to
+ensure such data is correct it is possible that bugs in either their code, or
+the game itself, could mean it is incorrect.  Listeners use such data at
+their own risk.
+
+Consumers can utilise the `$schemaRef` value to determine which Schema a
+particular message is for.  There is no need to validate the messages
+against the Schemas yourself, as that is performed on the EDDN Gateway.  
+Messages that do not pass the schema validation there are not forwarded to
+receivers.
+
+There is [example code](https://github.com/EDCD/EDDN/tree/master/examples)
+available for a variety of programming languages to help you get started.

examples/Java/SimpleSubscribe/pom.xml

@@ -0,0 +1,72 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+
+  <groupId>com.example</groupId>
+  <artifactId>SimpleJavaEDDNSubscribe</artifactId>
+  <version>1.0-SNAPSHOT</version>
+
+  <name>SimpleJavaEDDNSubscribe</name>
+
+  <properties>
+    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+    <maven.compiler.source>1.11</maven.compiler.source>
+    <maven.compiler.target>1.11</maven.compiler.target>
+  </properties>
+
+  <dependencies>
+    <dependency> 
+      <groupId>org.zeromq</groupId>
+      <artifactId>jeromq</artifactId>
+      <version>0.5.2</version>
+    </dependency>
+  </dependencies>
+
+  <build>
+    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
+      <plugins>
+        <!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
+        <plugin>
+          <artifactId>maven-clean-plugin</artifactId>
+          <version>3.1.0</version>
+        </plugin>
+        <!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
+        <plugin>
+          <artifactId>maven-resources-plugin</artifactId>
+          <version>3.0.2</version>
+        </plugin>
+        <plugin>
+          <artifactId>maven-compiler-plugin</artifactId>
+          <version>3.8.0</version>
+        </plugin>
+        <plugin>
+          <artifactId>maven-surefire-plugin</artifactId>
+          <version>2.22.1</version>
+        </plugin>
+        <plugin>
+          <artifactId>maven-jar-plugin</artifactId>
+          <version>3.0.2</version>
+        </plugin>
+        <plugin>
+          <artifactId>maven-install-plugin</artifactId>
+          <version>2.5.2</version>
+        </plugin>
+        <plugin>
+          <artifactId>maven-deploy-plugin</artifactId>
+          <version>2.8.2</version>
+        </plugin>
+        <!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
+        <plugin>
+          <artifactId>maven-site-plugin</artifactId>
+          <version>3.7.1</version>
+        </plugin>
+        <plugin>
+          <artifactId>maven-project-info-reports-plugin</artifactId>
+          <version>3.0.0</version>
+        </plugin>
+      </plugins>
+    </pluginManagement>
+  </build>
+</project>

examples/Java/SimpleSubscribe/src/main/java/org/eddn/examples/SimpleJavaEDDNSubscribe.java

@@ -0,0 +1,53 @@
+package org.eddn.examples;
+
+import java.util.zip.Inflater;
+
+import org.zeromq.SocketType;
+import org.zeromq.ZMQ;
+import org.zeromq.ZContext;
+
+public class SimpleJavaEDDNSubscribe {
+    private static final int MAX_MESSAGE_SIZE_KB = 200;
+    private static final String EDDN_SERVER = "tcp://eddn.edcd.io:9500";
+    private static ZContext context;
+
+    public static void main(String[] args) throws Exception {
+        context = new ZContext();    
+        String jsonMessage = getOneMessage();
+        System.out.println(jsonMessage);
+        context.close();
+    }
+
+    private static String getOneMessage() throws Exception {
+        byte[] deflatedMessage = receiveOneDeflatedMessage();
+        return inflateMessage(deflatedMessage);
+    }
+
+    private static byte[] receiveOneDeflatedMessage() {
+        ZMQ.Socket socket = getEDDNSubscriptionSocket();    
+        byte[] deflatedMessage = socket.recv();
+        return deflatedMessage;
+    }
+
+    private static ZMQ.Socket getEDDNSubscriptionSocket() {
+        ZMQ.Socket socket = context.createSocket(SocketType.SUB);
+        socket.connect(EDDN_SERVER);
+        
+        // need to subscribe to the empty topic to receive anything
+        socket.subscribe("");
+        return socket;
+    }
+
+    public static String inflateMessage(byte[] bytes) throws Exception {
+
+        Inflater decompresser = new Inflater();
+        decompresser.setInput(bytes);
+        byte[] result = new byte[MAX_MESSAGE_SIZE_KB * 1024];
+        int resultLength = decompresser.inflate(result);
+        decompresser.end();
+
+        // Decode the bytes into a String
+        String outputString = new String(result, 0, resultLength, "UTF-8");
+        return outputString;
+    }
+}

schemas/README-EDDN-schemas.md

@@ -1,172 +1,44 @@
 # EDDN Schemas Documentation
 
-## Introduction
+EDDN message Schemas are [JSON](https://www.json.org/json-en.html) files
+conforming to 'draft 04' of the [JSON Schema](https://json-schema.org/)
+specification.
 
-EDDN is a
-[zermoq](https://zeromq.org/) service to allow players of the game
-[Elite Dangerous](https://www.elitedangerous.com/), published
-by [Frontier Developments](https://www.frontier.co.uk/), to upload game data so
-that interested listeners can receive a copy.
+## Canonical location of Schema files
 
-EDDN accepts HTTP POST uploads in a defined format representing this game data
-and then passes it on to any interested listeners.
+For the EDDN Live service you should always be checking
+[the live version of the schemas, and their READMEs](https://github.com/EDCD/EDDN/tree/live/schemas).
+Any other version of the Schemas is not guaranteed to be synchronized with
+those actually running on the Live service.
 
----
-## Sources
+## Documentation of Schema files
 
-There are two sources of game data, both provided by the publisher of the game,
-Frontier Developerments.  They are both explicitly approved for use by 
-third-party software.
+The Schema files themselves are considered to be the canonical definition of
+the required, and allowed, contents of the relevant EDDN message.  There
+**MUST** be an accompanying README file, e.g. for `commodity-v3.0.json` there
+is also a `commodity-README.md` file in the project root `schemas/` directory.
 
-### Journal Files
+For more general documentation that all developers wanting to either Upload
+messages or Listen to the stream of messages from the Relay, please consult
+[the Developer documentation](../docs/Developers.md).
 
-On the PC version of the game, "Journal files" are written during any game
-session. These are in newline-delimited JSON format, with each line
-representing a single JSON object. Frontier Developments publishes
-documentation for the various events in their
-[Player Tools & API Discussions](https://forums.frontier.co.uk/forums/elite-api-and-tools/)
-forum.
+### Mandatory Schema file contents
 
-In general the documentation is made available in a file named something like:
+It is best to base any new Schema file on
+[the provided template](./TEMPLATES/journalevent-v1.0.json).  As per its
+contents all Schemas specify a top-level JSON Object with the data:
 
-    Journal_Manual-v<version>
-
-as both a MicroSoft word `.doc` file, or a `.pdf` file.  Historically the 
-use of `_` versus `-` in those filenames has varied.
-
-Consult the latest of these for documentation on individual events.  
-However, be aware that sometimes the documentation is in error, possibly due to
-not having been updated after a game client change.
-
-### Companion API (CAPI) data
-
-Frontier Developments provides an API to retrieve certain game data, even 
-without the game running.  Historically this was for use by its short-lived
-iOS "Companion" app, and was only intended to be used by that app. There was no
-public documentation, or even admission of its existence.
-
-Eventually, after some enterprising players had snooped the connections and
-figured out the login method and endpoints, Frontier Developments
-[allowed general use of this](https://forums.frontier.co.uk/threads/open-letter-to-frontier-developments.218658/page-19#post-3371472)
-.
-
-Originally the API authentication required being supplied with the email and
-password as used to login to the game (but at least this was over HTTPS).
-
-In late 2018 Frontier switched the authentication to using an oAuth2 flow,
-meaning players no longer need to supply their email and password to
-third-party sites and clients.
-
-As of October 2021 there has still never been any official documentation about
-the available endpoints and how they work. There is some
-[third-party documentation](https://github.com/Athanasius/fd-api/blob/main/docs/README.md)
-by Athanasius.
-
-It is *not* recommended to use CAPI data as the source as it's fraught with
-additional issues.  EDMarketConnector does so in order to facilitate
-obtaining data without the player needing to open the commodities screen.
-
-#### Detecting CAPI data lag
-
-When using the Companion API please be aware that the server that supplies this
-data sometimes lags behind the game - usually by a few seconds, sometimes by
-minutes. You MUST check in the data from the CAPI that the Cmdr is
-docked, and that the station and system names match those
-reported from the Journal before using the data for the commodity, outfitting
-and shipyard schemas:
-
-1. Retrieve the commander data from the `/profile` CAPI endpoint.
-2. Check that `commander['docked']` is true.  If not, abort.
-3. Retrieve the data from the `/market` and `/shipyard` CAPI endpoints.
-4. Compare the system and station name from the CAPI market data,
-   `["lastStarport"]["name"]` and `["lastSystem"]["name"]`,
-   to that from the last `Docked` or `Location` journal event.  If either does
-   not match then you MUST **abort**.  This likely indicates that the CAPI 
-   data is lagging behind the game client state and thus should not be used.
-
----
-
-## Uploading messages
-
-### Send only live data to the live schemas
-You MUST **NOT** send information from any non-live (e.g. alpha or beta)
-version of the game to the main schemas on this URL.
-
-You MAY send such to this URL so long as you append `/test` to the `$schemaRef`
-value, e.g.
-
-    "$schemaRef": "https://eddn.edcd.io/schemas/shipyard/2/test",
-
-You MUST also utilise these test forms of the schemas when first testing your
-code.
-
-There might also be a beta.eddn.edcd.io, or dev.eddn.edcd.io, service
-available from time to time as necessary, e.g. for testing new schemas or
-changes to existing ones.  Ask on the `#eddn` channel of the EDCD Discord 
-(see https://edcd.github.io/ for an invite link).
-
-Alternatively you could attempt
-[running your own test instance of EDDN](../docs/Running-this-software.md).
-
-### Sending data
-Messages sent to EDDN **MUST**:
-
-- Use the URL: `https://eddn.edcd.io:4430/upload/`.  Note the use of 
-  TLS-encrypted HTTPS.  A plain HTTP request will elicit a `400 Bad 
-  Request` response.
-- Use the HTTP 1.1 protocol.  HTTP/2 is not supported at this time.
-- Use a **POST** request, with the body containing the EDDN message.  No
-  query parameters in the URL are supported or necessary.
-
-The body of an EDDN message is a JSON object in UTF-8 encoding.  You SHOULD 
-set a `Content-Type` header of `applicaton/json`, and NOT any of:
-
-* `application/x-www-form-urlencoded`
-* `multipart/form-data`
-* `text/plain`
-
-For historical reasons URL form-encoded data *is* supported, **but this is 
-deprecated and no new software should attempt this method**.  We 
-purposefully do not further document the exact format for this.
-
-You *MAY* use gzip compression on the body of the message, but it is not 
-required.
-
-You should be prepared to handle all scenarios where sending of a message 
-fails:
-
-1. Connection refused.
-2. Connection timed out.
-3. Other possible responses as documented in
-   [Server responses](#server-responses).
-
-Carefully consider whether you should queue a 'failed' message for later 
-retry.  In particular, you should ensure that one 'bad' message does not 
-block other messages from being successfully sent.
-
-You **MUST** wait some reasonable time (minimum 1 minute) before retrying 
-any failed message.
-
-You **MUST NOT** retry any message that received a HTTP `400` or `426` code.
-An exception can be made if, **and only if**, *you have manually verified that 
-you have fixed the issues with it (i.e. updated the schema/version to a
-currently supported one and adjusted the data to fit that schema/version).*
-
-You **MAY** retry a message that initially received a `413` response (in 
-the hopes that the EDDN service admins decided to increase the maximum 
-allowed request size), but should not do so too quickly or in perpetuity.
-
-In general:
-
-- No data is better than bad data.
-- *Delayed* good data is better than degrading the EDDN service for others.
-
-### Format of uploaded messages
-Each message is a JSON object in UTF-8 encoding containing the following
-key+value pairs:
-
-1. `$schemaRef` - Which schema (including version) this message is for.
-2. `header` - Object containing mandatory information about the upload;
+1. `$schemaRef` - Which Schema (including version) this message is for.
+2. `$id` - The canonical URL for this schema once it is in live service.
+   1. Remember to have the version as in `journal/1` not `journal-v1.0`.
+   2. Do **NOT** end this with a `#` empty fragment.  This is
+  [documented](https://json-schema.org/draft/2020-12/json-schema-core.html#section-8.2.1)
+  as unnecessary.
+   3. Where there are two separate schemas for the same kind of data, but one
+     is for Journal-sourced, and the other for CAPI-sourced, you should have
+     the "filename" of the schema end with `_<source>`, e.g.
+     `fcmaterials_journal/1` and `fcmaterials_capi/1`.
+3. `header` - Object containing mandatory information about the upload;
     1. `uploaderID` - a unique ID for the player uploading this data.  
        Don't worry about privacy, the EDDN service will hash this with a key
        that is regularly changed so no-one knows who an uploader is in-game.
@@ -175,238 +47,62 @@ key+value pairs:
 
    Listeners MAY make decisions about whether to utilise the data in any
    message based on the combination of `softwareName` and `softwareVersion`.
-   
+
    **DO not** add `gatewaytimestamp` yourself. The EDDN Gateway will add
    this and will overwrite any that you provide, so don't bother.
 4. `message` - Object containing the data for this message. Consult the
    relevant README file within this documentation, e.g.
-   [codexentry-README.md](./codexentry-README.md). There are some general
-   guidelines [below](#contents-of-message).
+   [codexentry-README.md](./codexentry-README.md).
 
-For example, a shipyard message, version 2, might look like:
+Whilst currently (2022-11-23) the `gameversion` and `gamebuild` fields in the
+message `header` aren't yet mandatory in the sense of what's in the schema
+definitions, all Senders are **strongly encouraged** to send them.
 
-```JSON
-{
-  "$schemaRef": "https://eddn.edcd.io/schemas/shipyard/2",
-  "header": {
-    "uploaderID": "Bill",
-    "softwareName": "My excellent app",
-    "softwareVersion": "0.0.1"
-  },
-  "message": {
-    "systemName": "Munfayl",
-    "stationName": "Samson",
-    "marketId": 128023552,
-    "horizons": true,
-    "timestamp": "2019-01-08T06:39:43Z",
-    "ships": [
-      "anaconda",
-      "dolphin",
-      "eagle",
-      "ferdelance",
-      "hauler",
-      "krait_light",
-      "krait_mkii",
-      "mamba",
-      "python",
-      "sidewinder"
-    ]
-  }
-}
-```
+Indeed, the per-schema documentation, for every schema now states:
 
-### Contents of `message`
-Every message MUST comply with the schema its `$schemaRef` value cites.  
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
 
-Apart from short time windows during deployment of a new version the live 
-EDDN service should always be using
-[the schemas as present in the live branch](https://github.com/EDCD/EDDN/tree/live/schemas).
-So, be sure you're checking the live versions and not, e.g. those in the 
-`master` or other branches.
+### General EDDN message outline
 
 Each `message` object must have, at bare minimum:
 
-1. `timestamp` - string date and time in ISO8601 format. Whilst this
-   technically allows for any timezone to be cited you SHOULD provide this in
-   UTC, aka 'Zulu Time' as in the example above. You MUST ensure that you are
-   doing this properly. Do not claim 'Z' whilst actually using a local time
-   that is offset from UTC.
-   
-   If you are only utilising Journal-sourced data then simply using the 
-   value from there should be sufficient as the PC game client is meant to 
-   always be correctly citing UTC for this.  Indeed it has been observed, 
-   in the Odyssey 4.0.0.1002 client, that with the Windows clock behind UTC 
-   by 21 seconds both the in-game UI clock *and* the Journal event 
+1. `timestamp` - string date and time in ISO8601 format. 
+    1. Whilst this technically allows for any timezone to be cited you SHOULD
+      provide this in UTC, aka 'Zulu Time' as in the example above.
+      You MUST ensure that you are doing this properly. 
+      Do not claim 'Z' whilst actually using a local time that is offset from
+      UTC.
+    2. Historically we had never been explicit about if Senders should include
+      sub-second resolution in the timestamps, or if Listeners should be
+      prepared to accept such.  As of 2022-06-24 we are explicitly stating that
+      Senders **MAY** include sub-second resolution, and Listeners **MUST**
+      be prepared to accept such.
+
+   If you are only utilising Journal-sourced data then simply using the
+   value from there should be sufficient as the PC game client is meant to
+   always be correctly citing UTC for this.  Indeed it has been observed,
+   in the Odyssey 4.0.0.1002 client, that with the Windows clock behind UTC
+   by 21 seconds both the in-game UI clock *and* the Journal event
    timestamps are still properly UTC to the nearest second.
 
    Listeners MAY make decisions on accepting data based on this time stamp,
    i.e. "too old".
-2. At least one other key/value pair representing the data. In general there 
+2. Where the data is sourced from a Journal event please do preserve the
+   `event` key and value.  Yes, where we use an event-specific Schema this
+   might seem redundant, but it might aid an EDDN listener in streamlining
+   their code, and it does no harm.
+
+   Any new Schema based on Journal data **MUST** make `event` a required
+   property of the `message` dictionary.
+3. At least one other key/value pair representing the data. In general there
    will be much more than this. Consult the
-   [schemas and their documentation](./).
+   [Schemas and their documentation](./).
 
-Because the first versions of some schemas were defined when only the CAPI 
-data was available, before Journal files existed, many of the key names chosen
-in the schemas are based on the equivalent in CAPI data, not Journal events.
-This means ouy MUST rename many of the keys from Journal events to match the
-schemas.
+4. Please consult the advice pertaining to
+   [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags) and include them
+   whenever possible.
 
-EDDN is intended to transport generic data not specific to any particular Cmdr
-and to reflect only the data that every player would see in-game in station 
-services or the local map. To that end, uploading applications MUST ensure
-that messages do not contain any Cmdr-specific data (other than "uploaderID",
-the "horizons" flag, and the "odyssey" flag).
-
-The individual schemas will instruct you on various elisions (removals) to 
-be made to comply with this.
-
-Some of these requirements are also enforced by the schemas, and some things
-the schemas enforce might not be explicitly called out here.  So, **do**
-check what you're sending against the relevant schema(s) when making any 
-changes to your code.
-
-It is also advisable to Watch this repository on GitHub so as to be aware 
-of any changes to schemas.
-
-### Server responses
-There are three possible sources of HTTP responses when sending an upload 
-to EDDN.
-
-1. The reverse proxy that initially accepts the request.
-2. The python `bottle` module that the Gateway uses to process the 
-   forwarded requests.  This might object to a message before the actual 
-   EDDN code gets to process it at all.
-3. The actual EDDN Gateway code.
-
-Once a message has cleared the EDDN Gateway then there is no mechanism for any 
-further issue (such as a message being detected as a duplicate in the 
-Monitor downstream of the Gateway) to be reported back to the sender.
-
-To state the obvious, if there are no issues with a request then an HTTP 
-200 response will be received by the sender.  The body of the response 
-should be the string `OK`.
-
-#### Reverse Proxy responses
-In addition to generic "you typoed the URL" and other such "you just didn't 
-make a valid request" responses you might experience the following:
-
-1. `408` - `Request Timed Out` - the sender took too long to make/complete 
-   its request and the reverse proxy rejected it as a result.
-2. `503` - `Service Unavailable` - the EDDN Gateway process is either not 
-   running, or not responding.
-3. `400` - `Bad Request` - if you attempt to use plain HTTP, rather than
-   HTTPS.
-
-#### bottle responses
-1. `413` - `Payload Too Large` - `bottle` enforces a maximum request size 
-   and the request exceeds that.  As of 2022-01-07 the limit is 1MiB, and 
-   pertains to the plain-text size, not after gzip compression if used.
-   To verify the current limit check for the line that looks like:
-
-      ```
-      bottle.BaseRequest.MEMFILE_MAX = 1024 * 1024 # 1MiB, default is/was 100KiB
-      ```
-   
-   in
-   [src/eddn/Gateway.py](https://github.com/EDCD/EDDN/blob/live/src/eddn/Gateway.py),
-   as added in
-   [commit  0e80c76cb564771465f61825e694227dcc3be312](https://github.com/EDCD/EDDN/commit/0e80c76cb564771465f61825e694227dcc3be312).
-
-#### EDDN Gateway responses
-For all failures the response body will contain text that begins `FAIL: `.  Currently two different HTTP status codes are utilised:
-
-1. `400` - `Bad Request` - This indicates something wrong with the request 
-   body.  Possibly due to a format issue (compression, form encoding), or 
-   the actual content of the EDDN message:
-   1. `FAIL: zlib.error: <detail>` - A failure to decompress a message that 
-      claimed to be compressed.
-
-   2. `FAIL: Malformed Upload: <detail>` - the message appeared to be 
-      form-encoded, but either the format was bad or there was no `data` 
-      key.
-
-   3. `FAIL: JSON parsing: <detail>` - the 
-       message couldn't be parsed as valid JSON.  e.g.
-
-   ```
-   FAIL: JSON parsing: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
-   ```
-   
-   4. `FAIL: Schema Validation: <detail>` - the message failed to validate 
-      against the cited schema. e.g.
-   
-   ```
-   FAIL: Schema Validation: [<ValidationError: "'StarPos' is a required property">]
-   ```
-   
-   The exact detail will be very much dependent on both the schema the 
-   message cited and the contents of the message that failed to pass the 
-   validation.
-
-   In particular, if the message contains a key that is tagged 'disallowed' in 
-   the schema the response will look like:
-
-   ```
-   FAIL: Schema Validation: "[<ValidationError: "{'type': ['array', 'boolean', 'integer', 'number', 'null', 'object', 'string']} is not allowed for 'BadVALUE'">]"
-   ```
-   This is due to the use of a JSON schema stanza that says "don't allow 
-   any valid type for the value of this key" to trigger the error for such
-   disallowed keys.
-
-   Note how the **value** for the disallowed key is cited, not the key *name*
-   itself.  This is a limitation of how the `jsonschema` python module
-   reports errors, and we are
-   [hoping to improve this](https://github.com/EDCD/EDDN/issues/163).
-
-2. `426` - `Upgrade Required` - This indicates that the cited schema, or 
-   version thereof, is outdated.  The body of the response will be:
-
-   ```
-   FAIL: Oudated Schema: The schema you have used is no longer supported. Please check for an updated version of your application.
-   ```
-   The wording here is aimed at users of applications that send messages 
-   over EDDN.  If you're the developer of such an application then 
-   obviously you need to update your code to use a currently supported 
-   schema and version thereof.
-
-
-There shouldn't be any other variants of a 'FAIL' message.  If you find
-any then please
-[open an issue on GitHub](https://github.com/EDCD/EDDN/issues/new)
-with as much detail as possible so that we can update this documentation.
-
-## Receiving messages
-
-EDDN provides a continuous stream of information from uploaders. To use this
-data you'll need to connect to the stream using ZeroMQ (a library is probably
-available for your language of choice).
-
-The URL for the live Relay is:
-
-    tcp://eddn.edcd.io:9500
-
-Depending on the programming language and library used, you might need
-to explicitly subscribe to an empty topic, `''`, in order to receive
-anything.
-
-Unfortunately at this time we're using an old version of
-ZeroMQ which does not support server-side subscription filtering.  So by
-all means specify a more specific topic filter if it suits your needs,
-but realise that the dropping of not-matching messages will happen at
-your end, not the server.
-
-Once you've connected to that you will receive messages.  To access the 
-data you will first need to zlib-decompress each message.  Then you will 
-have a textual JSON object as per the schemas.
-
-In general, check the guidance for [Uploading messages](#uploading-messages)
-for the expected format of the messages.
-
-Consumers can utilise the `$schemaRef` value to determine which schema a 
-particular message is for.  There is no need to validate the messages 
-against the schemas yourself, as that is performed on the EDDN Gateway.  
-Messages that do not pass the schema validation there are not forwarded to 
-receivers.
-
-There is [example code](https://github.com/EDCD/EDDN/tree/master/examples)
-available for a variety of programming languages to help you get started.
+Where a key has to be renamed this will be specified in the Schema through a
+`renamed` property on the object in question.

schemas/TEMPLATES/journalevent-README.md

@@ -0,0 +1,125 @@
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+
+!!! STOP !!!
+
+!! These are not valid MarkDown comments.
+
+!! You MUST read, comply with, and then remove all of these lines to have a
+
+!! valid Schema README file.
+
+1. Be careful if using any existing Schema, or its README, as reference for a
+   new Schema.  Some of them will have some departures from best practices for
+   historical reasons.  It's usually not easy to co-ordinate all Listeners and
+   Senders smoothly updating to a new version of a Schema, so less than ideal
+   requires/optional/disallowed definitions might be present in older Schemas.
+
+   [The main Schema documentation](../README-EDDN-schemas.md) is the canonical
+   authority.
+2. Replace all instances of `NewJournalEvent` with the name of the actual
+   Journal event's Schema you are documenting.  This should have the case
+   preserved as per how it appears in actual game Journal files.
+3. Replace all instances of `newjournalevent` with the lower-case folded
+   version of this Schema's Journal event name.
+4. For new Journal-based schemas no key renames should be necessary.
+
+   If there are no renames of key names for this Schema, then remove the
+   `Key Renames` section.
+
+   Where such renames *are* required do **NOT** attempt to list them all here.
+   That would just require updating them both here and in the actual Schema.
+
+   If there are any, then the affected property object should contain a key
+   named `renamed` with its value being the original key name in the source
+   data, e.g. in the commodity/3 schema a Journal `StarSystem` is renamed
+   to `systemName` so we have:
+
+      ```json
+        "message": {
+            "type"                  : "object",
+            "additionalProperties"  : false,
+            "required"              : [ "systemName", "stationName", "marketId", "timestamp", "commodities" ],
+            "properties"            : {
+                "systemName": {
+                    "type"      : "string",
+                    "renamed"   : "StarSystem",
+                    "minLength" : 1
+                },
+
+      ```
+
+5. Do **NOT** remove the `horizons and odyssey flags` section.  It is
+   mandatory that they are allowed (but are optional) in any Journal-based
+   EDDN Schema.
+
+6. If both:
+   1. either the source Journal event contains information that includes the
+      System name (possibly as `StarSystem` or `SystemName`), **OR** the source
+      data contains a `SystemAddress` value,
+   2. and a `StarPos` array is *not already present* in the source data.
+   
+   then you MUST include the `StarPos` section in `Augmentations` and add
+   `StarPos` to the `required` message properties in the Schema file.
+
+   If neither key is in the source data then remove the `StarPos` section from
+   this document and the Schema file.
+
+7. Do **NOT** add an 'Elisions'/'Removals' section.  Leave the Schema as the
+   sole reference for any data that is in the source but should not be in the
+   final EDDN message.
+
+The line:
+
+    # EDDN NewJournalEvent Schema
+
+below should ultimately be the first line in this file, after required edits.
+
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+
+# EDDN NewJournalEvent Schema
+
+## Introduction
+Here we document how to take data from an ED `NewJournalEvent` Journal 
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The data source for this schema is the ED Journal event `NewJournalEvent`.
+
+### Key Renames
+Some key names in this Schema are different from how they appear in the source
+Journal data.  Look for keys where the object contains a `renamed` key - the
+value is what the name would have been in the source Journal data.
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### StarSystem
+You MUST add a `StarSystem` string containing the name of the system from the 
+last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
+#### StarPos
+You MUST add a `StarPos` array containing the system co-ordinates from the 
+last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
+## Listeners
+The advice above for [Senders](#senders), combined with the actual Schema file
+*should* provide all the information you need to process these events.

schemas/TEMPLATES/journalevent-v1.0.json

@@ -0,0 +1,113 @@
+# This is not a valid JSON file
+# Removed all the 'comments' as you are sure you have complied with their
+# instructions!
+#
+# 1. Replace all instances of newjournalevent with the lower-case folded name
+#    of the Journal event this schema is for.
+# 2. Replace all instances of NewJournalEvent with the name of the journal
+#   event this schema is for, as it appears in the Journal.  Specifically,
+#   you must conserve the case.
+# 3. Leave the 'header' dictionary exactly as it is, it is mandatory.
+# 4. Add any additional schema-mandatory message properties to the 'required'
+#    array.
+# 5. If, and ONLY IF, you have good reason to believe there might be additional
+#    valid keys in the source data, change the 'additionalProperties' value
+#    to 'true'.
+#    EMPHASIS: You should really know about all of the possible keys and their
+#              values and be defining them, possibly as optional (not listed in
+#              'required') properties.  If needs be document, in the README for
+#              this Schema, that all but the defined properties should be
+#              elided when constructing a message.
+# 6. The 'horizons' and 'odyssey' properties in 'message' MUST BE RETAINED.
+#    Any message based on Journal data should be able to add them as
+#    appropriate.  Note that they *are optional in the resulting message*.
+#    You MUST NOT add these to the 'required' list, as the LoadGame Journal
+#    event is not guaranteed to contain either, depending on which client
+#    (base, horizons, odyssey) the player is running.
+# 7. 'some_disallowed_key' demonstrates how to specify that a key (and thus its
+#    value) is not allowed in this schema.  Note the **MANDATORY** description.
+# 8. Note the trailing comma after the 'some_disallowed_key' entry in
+#    'messages'.  If all you did was remove these comments you would NOT have
+#    a valid JSON file.  You should be adding **at least** one additional
+#    pertinent 'message' key:value definition.
+# 9. Consult the journalevent-README.md file, particularly the section about
+#    `StarPos` and remove that section of this file if it is not needed.
+# 10. You MUST NOT remove the 'disallowed' key, or edit its value.  The purpose
+#    of this is to be cited as the allowed value on a disallowed key.  As it
+#    defines that the only valid type for the value is **not** any of the
+#    possible JSON types, its effect is to disallow any value for the key, and
+#    as a result disallow the key.
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/newjournalevent/1",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "NewJournalEvent" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Boolean value copied from the Journal LoadGame event, when it is present there."
+                },
+                "odyssey": {
+                    "type"          : "boolean",
+                    "description"   : "Boolean value copied from the Journal LoadGame event, when it is present there."
+                },
+                "StarPos": {
+                    "type"          : "array",
+                    "items"         : { "type": "number" },
+                    "minItems"      : 3,
+                    "maxItems"      : 3,
+                    "description"   : "Must be added by the sender"
+                },
+                "ExampleRenamedKey" : {
+                    "type"          : "string",
+                    "renamed"       : "SomeOtherKey"
+                }
+                "some_disallowed_key" {
+                    "$ref"          : "#/definitions/disallowed",
+                    "description"   : "MANDATORY brief description of why this key must be removed from source data"
+                },
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/approachsettlement-README.md

@@ -0,0 +1,79 @@
+# EDDN ApproachSettlement Schema
+
+## Introduction
+Here we document how to take data from an ED `ApproachSettlement` Journal 
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The primary data source for this schema is the ED Journal event 
+`ApproachSettlement`.
+
+### MarketID
+Whilst the `MarketID` property is not in the required list **YOU MUST
+ABSOLUTELY SEND THIS WHEN IT IS PRESENT IN THE SOURCE DATA**.
+
+The only reason it is optional is that there are `ApproachSettlement`
+Journal events for things like visitor beacons that do not have a market, and
+thus no MarketID.
+
+Examples:
+
+```json
+{
+    "timestamp":"2022-02-18T14:33:35Z",
+    "event":"ApproachSettlement",
+    "Name":"Battlegroup's Disappearance",
+    "SystemAddress":1109989017963,
+    "BodyID":8,
+    "BodyName":"Alioth 1 a",
+    "Latitude":59.972752,
+    "Longitude":-84.506294
+},
+{
+    "timestamp": "2022-02-18T15:02:04Z",
+    "event": "ApproachSettlement",
+    "Name": "$Ancient:#index=1;",
+    "Name_Localised": "Ancient Ruins (1)",
+    "SystemAddress": 3515254557027,
+    "BodyID": 13,
+    "BodyName": "Synuefe XR-H d11-102 1 b",
+    "Latitude": -46.576923,
+    "Longitude": 133.985107
+},
+```
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
+#### StarSystem
+
+You MUST add a StarSystem key/value pair representing the name of the system
+this event occurred in. Source this from either Location, FSDJump or
+CarrierJump as appropriate.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
+#### StarPos
+You MUST add a `StarPos` array containing the system co-ordinates from the
+last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**

schemas/approachsettlement-v1.0.json

@@ -0,0 +1,144 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/approachsettlement/1#",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event", "StarSystem", "StarPos", "SystemAddress", "Name", "BodyID", "BodyName", "Latitude", "Longitude" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "ApproachSettlement" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Whether the sending Cmdr has a Horizons pass."
+                },
+                "odyssey": {
+                    "type"      : "boolean",
+                    "description" : "Whether the sending Cmdr has an Odyssey expansion."
+                },
+                "StarSystem": {
+                    "type"          : "string",
+                    "minLength"     : 1,
+                    "description"   : "Must be added by the sender"
+                },
+                "StarPos": {
+                    "type"          : "array",
+                    "items"         : { "type": "number" },
+                    "minItems"      : 3,
+                    "maxItems"      : 3,
+                    "description"   : "Must be added by the sender"
+                },
+                "StationGovernment": {
+                    "type"          : "string"
+                },
+                "StationAllegiance": {
+                    "type"          : "string"
+                },
+                "StationEconomies": {
+                    "type"          : "array",
+                    "items"         : {
+                       "type"      : "object",
+                        "additionalProperties"  : false,
+                        "properties": {
+                            "Name": {
+                                "type"          : "string"
+                            },
+                            "Proportion": {
+                                "type"          : "number"
+                            }
+                        },
+                       "patternProperties"     : {
+                           "_Localised$"       : { "$ref" : "#/definitions/disallowed" }
+                       }
+                    }
+                },
+                "StationFaction": {
+                   "type"      : "object",
+                   "additionalProperties"  : false,
+                   "properties": {
+                       "Name": {
+                           "type"          : "string"
+                       },
+                       "FactionState": {
+                           "type"          : "string"
+                       }
+                   }
+                },	
+                "StationServices": {
+                    "type"          : "array",
+                    "items"         : { "type": "string" }
+                },
+                "StationEconomy": {
+                    "type"          : "string"
+                },	
+                "SystemAddress": {
+                    "type"          : "integer"
+                },
+                "Name"             : {
+                    "type"          : "string",
+                    "description"   : "Name of settlement"
+                },
+                "MarketID": {
+                    "type"          : "integer"
+                },		
+                "BodyID": {
+                    "type"          : "integer"
+                },
+                "BodyName": {
+                    "type"          : "string"
+                },	
+                "Latitude": {
+                    "type"          : "number"
+                },
+                "Longitude": {
+                    "type"          : "number"
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/blackmarket-README.md

@@ -20,17 +20,9 @@ documentation for a schema such as this.
 The primary data source for this schema is the ED Journal event `MarketSell`.
 
 ### Key Renames
-#### name
-Due to how the EDDN schema is defined the `Type` key/value should
-have the key renamed to `name`.
-
-#### prohibited
-Due to how the EDDN schema is defined the `IllegalGoods` key/value should
-have the key renamed to `prohibited`.
-
-#### marketID
-The Journal documentation says this is `MarketID`, but in the schema the 
-`m` is lower case.
+Some key names in this Schema are different from how they appear in the source
+Journal data.  Look for keys where the object contains a `renamed` key - the
+value is what the name would have been in the source Journal data.
 
 ### Elisions
 You MUST remove the following key/value pairs from the data:
@@ -41,15 +33,13 @@ You MUST remove the following key/value pairs from the data:
   - `BlackMarket` - Because we're using this schema, so this is un-necessary. 
 
 ### Augmentations
-#### horizons flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
-
-#### odyssey flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
 
 #### systemName
 The star system name for where this market is.   Use the `StarSystem` value
 from the prior `Docked` or `Location` event.
 
 #### stationName
-From the `StationName` value on the prior `Docked` or `Location` event.
+From the `StationName` value on the prior `Docked` or `Location` event.

schemas/blackmarket-v1.0.json

@@ -44,7 +44,8 @@
                     "minLength"     : 1
                 },                
                 "marketId": {
-                    "type"          : "integer"
+                    "type"          : "integer",
+                    "renamed"       : "MarketID"
                 },
                 "timestamp": {
                     "type"          : "string",
@@ -52,6 +53,7 @@
                 },
                 "name": {
                     "type"          : "string",
+                    "renamed"       : "Type",
                     "minLength"     : 1,
                     "description"   : "Commodity name as returned by the MarketSell entry in the Journal"
                 },
@@ -61,6 +63,7 @@
                 },
                 "prohibited": {
                     "type"          : "boolean",
+                    "renamed"       : "IllegalGoods",
                     "description"   : "Whether the commodity is prohibited at this station"
                 }
             }

schemas/codexentry-README.md

@@ -7,40 +7,40 @@ properly structure it for sending to EDDN.
 Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general 
 documentation for a schema such as this.
 
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
 ## Senders
 The primary data source for this schema is the ED Journal event `CodexEntry`.
 
-### Elisions
-You MUST remove any key where the key name ends in 
-`_Localised`.
-
-You MUST remove the two keys `IsNewEntry` and `NewTraitsDiscovered`.
-
 ### Augmentations
-#### horizons flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
 
-#### odyssey flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
 
 #### StarPos
-You MUST **add** a `StarPos` key with value of type `array` containing the 
-galaxy co-ordinates of the system.  You will need to have obtained these 
-from prior event(s) upon the player arriving, or logging into, the system.
+You MUST add a `StarPos` array containing the system co-ordinates from the
+last `FSDJump`, `CarrierJump`, or `Location` event.
 
-e.g. if the system is `Alpha Centauri`:
-```json
-    "StarPos": [3.03125, -0.09375, 3.15625]
-```
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
 
 #### BodyID and BodyName
 You SHOULD attempt to track the BodyName and BodyID where the player is 
 and add keys/values for these.
 
 You MUST track `BodyName` both from Status.json *and* also from some
-[Journal](./README-EDDN-schemas.md#journal-files)
-events in order to cross-check it before using the `BodyID` from 
-[Journal](./README-EDDN-schemas.md#journal-files) events.
+Journal events in order to cross-check it before using the `BodyID` from 
+Journal events.
 
 The following is correct as of game version 4.0.0.801 (Odyssey initial 
 release, Update 7, plus one patch).
@@ -103,7 +103,7 @@ e.g. for `Bestia A 2 a`
 If you cannot properly obtain the values for `BodyName` or `BodyID` then 
 you MUST NOT include them.
 
-## Receivers
+## Listeners
 
 As per ['BodyID and BodyName'](#bodyid-and-bodyname) above be aware that 
 you are not guaranteed to receive these values for any given event.  Some 

schemas/codexentry-v1.0.json

@@ -17,6 +17,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/commodity-README.md

@@ -7,6 +7,14 @@ properly structure it for sending to EDDN.
 Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
 documentation for a schema such as this.
 
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
 ## Senders
 The primary data source for this schema is the ED Journal event `Market`, 
 and the additional file, `Market.json`, that it signals the writing of.
@@ -24,18 +32,12 @@ See [Using CAPI data](#using-capi-data) below.
 So, as per the schema, do include it if available.
 
 ### Key Renames
-Many of the key names have a different case defined in this schema, make 
-sure you are renaming them as appropriate.
-
-#### StarSystem to systemName
-Rename the `StarSystem` key name to `systemName`.
+Some key names in this Schema are different from how they appear in source
+Journal data.  Look for keys where the object contains a `renamed` key - the
+value is what the name would have been in the source Journal data.  The names
+used are as found in the CAPI source data.
 
 ### Elisions
-#### Remove _Localised key/values
-All keys whose name ends with `_Localised`, i.e. the `Name_Localised`
-key/values in Items.
-
-#### Other Elisions
 You MUST remove the following key/value pairs from the data:
 
 - `StationType` key/value.
@@ -49,16 +51,23 @@ In the list of commodites:
   Limpets - not purchasable in station market) or a *non-empty*`"legality":` 
   string (not normally traded at this station market).
 
+If the data is sourced from the journal folder:
+- Remove the `$` prefix and `_name;` suffix from the `Name` field.
+- As the Journal Market.json doesn't contain `economies` or `prohibited` data, 
+  leave these entirely out of the message.  You **MUST NOT** send empty lists.
+
 #### Item Category
-Remove not only the `Category_Localised` key/value, as above, but also the
-`Category` key/value pair from each Item.
+Remove not only the `Category_Localised` key:values, but also the
+`Category` key:value pair from each Item.
 
 ### Augmentations
-#### horizons flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
 
-#### odyssey flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
 
 ### Using CAPI data
 It is *not* recommended to use CAPI data as the source as it's fraught with 
@@ -66,10 +75,10 @@ additional issues.  EDMarketConnector does so in order to facilitate
 obtaining data without the player needing to open the commodities screen.
 
 Please read
-[the guidance on checking for CAPI lag](README-EDDN-schemas.md#detecting-capi-data-lag)
+[the guidance on checking for CAPI lag](../docs/Developers.md#detecting-capi-data-lag)
 before utilising CAPI data for EDDN messages.
 
-Note that CAPI `/market` data will sometimes have the `StatusFlasg` per 
+Note that CAPI `/market` data will sometimes have the `statusFlasg` per 
 item, which are defined as optional in this schema (because they're not in 
 the Market.json data).  You SHOULD include this data in your message if 
 using CAPI as the source.
@@ -91,4 +100,4 @@ any of the listed ships or modules have a `sku` value of
 #### CAPI odyssey flag
 Unfortunately there is no method to be *certain* of this from CAPI data, so 
 you will have to trust in the system/station name check and use the value 
-from the Journal `LoadGame` event.
+from the Journal `LoadGame` event.

schemas/commodity-v3.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->gameversion, else LoadGame->gameversion, else 'CAPI-market', else ''."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->build, else LoadGame->build, else 'CAPI-market', else ''."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },
@@ -36,14 +44,25 @@
             "properties"            : {
                 "systemName": {
                     "type"      : "string",
+                    "renamed"   : "StarSystem",
                     "minLength" : 1
                 },
                 "stationName": {
                     "type"      : "string",
+                    "renamed"   : "StationName",
                     "minLength" : 1
                 },                
+                "stationType": {
+                    "type"      : "string",
+                    "renamed"   : "StationType"
+                },
+                "carrierDockingAccess": {
+                    "type"      : "string",
+                    "renamed"   : "CarrierDockingAccess"
+                },                
                 "marketId": {
-                    "type"      : "integer"
+                    "type"      : "integer",
+                    "renamed"   : "MarketID"
                 },
                 "horizons": {
                     "type"      : "boolean",
@@ -67,31 +86,39 @@
                         "properties"            : {
                             "name": {
                                 "type"          : "string",
+                                "renamed"       : "Name",
                                 "minLength"     : 1,
                                 "description"   : "Symbolic name as returned by the Companion API"
                             },
                             "meanPrice": {
-                                "type"          : "integer"
+                                "type"          : "integer",
+                                "renamed"       : "MeanPrice"
                             },
                             "buyPrice": {
                                 "type"          : "integer",
+                                "renamed"       : "BuyPrice",
                                 "description"   : "Price to buy from the market"
                             },
                             "stock": {
-                                "type"          : "integer"
+                                "type"          : "integer",
+                                "renamed"       : "Stock"
                             },
                             "stockBracket": {
-                                "$ref"          : "#/definitions/levelType"
+                                "$ref"          : "#/definitions/levelType",
+                                "renamed"       : "StockBracket"
                             },
                             "sellPrice": {
                                 "type"          : "integer",
+                                "renamed"       : "SellPrice",
                                 "description"   : "Price to sell to the market"
                             },
                             "demand": {
-                                "type"          : "integer"
+                                "type"          : "integer",
+                                "renamed"       : "Demand"
                             },
                             "demandBracket": {
-                                "$ref"          : "#/definitions/levelType"
+                                "$ref"          : "#/definitions/levelType",
+                                "renamed"       : "DemandBracket"
                             },
                             "statusFlags": {
                                 "type"          : "array",
@@ -101,6 +128,18 @@
                                     "type"          : "string",
                                     "minLength"     : 1
                                 }
+                            },
+                            "Producer": {
+                                "$ref"          : "#/definitions/disallowed",
+                                "description"   : "Not present in CAPI data, so removed from Journal-sourced data"
+                            },
+                            "Rare" : {
+                                "$ref"          : "#/definitions/disallowed",
+                                "description"   : "Not present in CAPI data, so removed from Journal-sourced data"
+                            },
+                            "id": {
+                                "$ref"          : "#/definitions/disallowed",
+                                "description"   : "Not wanted for historical reasons?"
                             }
                         }
                     }
@@ -135,6 +174,7 @@
         }
     },
     "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } },
         "levelType": {
             "enum"          : [0, 1, 2, 3, ""],
             "description"   : "Note: A value of \"\" indicates that the commodity is not normally sold/purchased at this station, but is currently temporarily for sale/purchase"

schemas/dockingdenied-README.md

@@ -0,0 +1,42 @@
+# EDDN DockingDenied Schema
+
+## Introduction
+Here we document how to take data from an ED `` Journal 
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The primary data source for this schema is the ED Journal event 
+`DockingDenied`.
+
+Examples:
+
+```json
+{
+    "timestamp":"2022-06-10T10:09:41Z",
+    "event":"DockingDenied",
+    "Reason":"RestrictedAccess",
+    "MarketID":3706117376,
+    "StationName":"V7G-T1G",
+    "StationType":"FleetCarrier"
+}
+```
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.

schemas/dockingdenied-v1.0.json

@@ -0,0 +1,82 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/dockingdenied/1#",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event", "MarketID", "StationName", "Reason" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "DockingDenied" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Whether the sending Cmdr has a Horizons pass."
+                },
+                "odyssey": {
+                    "type"      : "boolean",
+                    "description" : "Whether the sending Cmdr has an Odyssey expansion."
+                },
+                "MarketID": {
+                    "type"          : "integer"
+                },		
+                "StationName": {
+                    "type"          : "string",
+                    "description"   : "Name of station"
+                },
+                "StationType": {
+                    "type"          : "string",
+                    "description"   : "Type of station"
+                },
+                "Reason": {
+                    "type"          : "string",
+                    "description"   : "Reason docking was denied"
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/dockinggranted-README.md

@@ -0,0 +1,42 @@
+# EDDN DockingGranted Schema
+
+## Introduction
+Here we document how to take data from an ED `` Journal 
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The primary data source for this schema is the ED Journal event 
+`DockingGranted`.
+
+Examples:
+
+```json
+{
+    "timestamp":"2023-10-01T14:56:34Z",
+    "event":"DockingGranted",
+    "LandingPad":41,
+    "MarketID":3227312896,
+    "StationName":"Evans Horizons",
+    "StationType":"Coriolis"
+}
+```
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.

schemas/dockinggranted-v1.0.json

@@ -0,0 +1,82 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/dockinggranted/1#",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event", "MarketID", "StationName" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "DockingGranted" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Whether the sending Cmdr has a Horizons pass."
+                },
+                "odyssey": {
+                    "type"      : "boolean",
+                    "description" : "Whether the sending Cmdr has an Odyssey expansion."
+                },
+                "MarketID": {
+                    "type"          : "integer"
+                },		
+                "StationName": {
+                    "type"          : "string",
+                    "description"   : "Name of station"
+                },
+                "StationType": {
+                    "type"          : "string",
+                    "description"   : "Type of station"
+                },
+                "LandingPad": {
+                    "type"          : "integer",
+                    "description"   : "Pad number Cmdr was granted landing to"
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/fcmaterials_capi-README.md

@@ -0,0 +1,82 @@
+# EDDN FCMaterials Schema
+
+## Introduction
+This is the documentation for how to take data from an the ED CAPI `/market`
+endpoint and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The data source for this schema is the `/market` ED CAPI endpoint.  You only
+want selected parts of the full data returned for this schema.
+
+You **MUST NOT** construct the message by starting with the entirety of the
+CAPI data and then removing everything but what you need.  That risks Frontier
+adding more data to the endpoint and your `fcmaterials_capi` messages being
+rejected as invalid.  Instead, construct the message content by setting just
+the data that is necessary.
+
+Your `message` object **MUST**:
+1. Have an `"event":"FCMaterials"` member to aid Listeners who pass this
+   through a "usually from the Journal" code path.
+2. Set a `"MarketID"` key with the value from `"id"` in the CAPI data.
+3. Set a `"CarrierID"` key with the value from the `"name"` in the CAPI data.
+4. Set the `"Items"` key's contents directly from the `/market` -> `orders`
+   -> `onfootmicroresources` CAPI data.
+5. Remove any data where the key is `"locName"` from the `"Items"` data.
+
+You **MUST NOT**:
+1. Attempt to set a `"CarrierName"` from any source, the `CarrierID` is
+   sufficient.
+
+### Example algorithm
+
+1. Make a CAPI `/market` query.
+2. Set `event`, `MarketID` and `CarrierID` as outlined above.
+3. Set `Items` value to the data in `orders.onfootmicroresources`.
+4. Process the contents of `Items`, removing any data with a key of `locName`.
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+You **SHOULD** set these flags from the Journal `FileHeader` data if you are
+reasonably sure you have a live game session against which you are performing
+CAPI queries.
+You **MUST NOT** set them otherwise, as e.g. the player could be active in
+the game on another computer, using a different game mode and the CAPI data
+will be for that game mode.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
+## Listeners
+The advice above for [Senders](#senders), combined with the actual Schema file
+*should* provide all the information you need to process these events.
+
+Do note that the data source for this is the CAPI, and as such the data is not
+the same as for the `fcmaterials_journal` schema:
+
+1. There is no good source of `CarrierName` in CAPI `/market` endpoint data, so 
+   that is not included.
+2. The `sales` and `purchases` values do **not** contain the same form of data.
+3. The `sales` member of `Items` will be `[]` if there are no sales orders, but
+   when there are orders:
+    1. It will be an object/dictionary.
+    2. The keys are the commodity ID.
+    3. The value of that key is the rest of the data for that sales order.
+4. The `purchases` value:
+    1. Is always an array, unlike `sales`.
+    2. As a consequence does **not** provide the commodity id at all, only
+      the name.

schemas/fcmaterials_capi-v1.0.json

@@ -0,0 +1,152 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/fcmaterials_capi/1",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "Value of 'CAPI-market' if possible, else empty string."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "Value of 'CAPI-market' if possible, else empty string."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event", "MarketID", "CarrierID", "Items" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "FCMaterials" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Boolean value copied from the Journal LoadGame event, when it is present there."
+                },
+                "odyssey": {
+                    "type"          : "boolean",
+                    "description"   : "Boolean value copied from the Journal LoadGame event, when it is present there."
+                },
+
+                "MarketID": {
+                    "type"          : "integer"
+                },
+
+                "CarrierID": {
+                    "type"          : "string",
+                    "minLength"     : 1
+                },
+
+                "Items": {
+                    "properties": {
+                        "purchases": {
+                            "type": "array",
+                            "items": {
+                                "type": "object",
+                                "required": [ "name", "price", "outstanding", "total" ],
+                                "additionalProperties": false,
+                                "properties": {
+                                    "name": {
+                                        "type": "string",
+                                        "minLength": 1
+                                    },
+                                    "locName": {
+                                        "$ref": "#/definitions/disallowed"
+                                    },
+                                    "outstanding": {
+                                        "type": "integer"
+                                    },
+                                    "price": {
+                                        "type": "integer"
+                                    },
+                                    "total": {
+                                        "type": "integer"
+                                    }
+                                }
+                            }
+                        },
+                        "sales": {
+                            "anyOf": [
+                                {
+                                    "type": "array",
+                                    "$comment": "If there are no items then sales is an empty array",
+                                    "minItems": 0,
+                                    "maxItems": 0
+                                },
+                                {
+                                    "type": "object",
+                                    "$comment": "If there ARE items then sales is an object, *NOT* an array",
+                                    "patternProperties": {
+                                        "^[0-9]+$": {
+                                            "type"      : "object",
+                                            "required"  : [ "id", "name", "price", "stock" ],
+                                            "additionalProperties": false,
+                                            "properties": {
+                                                "id"    : {
+                                                    "type"  : "integer"
+                                                },
+                                                "name"  : {
+                                                    "type"  : "string",
+                                                    "minLength": 1
+                                                },
+                                                "locName"   : {
+                                                    "$ref": "#/definitions/disallowed"
+                                                },
+                                                "price" : {
+                                                    "type": "integer"
+                                                },
+                                                "stock": {
+                                                    "type": "integer"
+                                                }
+                                            }
+                                        }
+                                    }
+                                }
+                            ]
+                        }
+                    }
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed": {
+            "not" : {
+                "type": [
+                    "array", "boolean", "integer", "number", "null", "object", "string"
+                ]
+            }
+        }
+    }
+}

schemas/fcmaterials_journal-README.md

@@ -0,0 +1,42 @@
+# EDDN FCMaterials Schema
+
+## Introduction
+This is the documentation for how to take data from an ED `FCMaterials.json`
+file and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The data source for this schema is the file `FCMaterials.json`.  That it has
+been freshly written is signalled by the ED Journal event `FCMaterials`.
+**NB: This schema is not, currently, for sending CAPI `/market`-sourced data
+about these materials.**
+
+So, monitor the Journal as normal, and when you see a `FCMaterials` event open
+the `FCMaterials.json` file for reading, read it, and close it again.  Use the
+data you got from reading this file, not merely the Journal event.
+
+Your `message` should primarily be the contents of this file, with the addition
+of any augmentations, as noted below.
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
+## Listeners
+The advice above for [Senders](#senders), combined with the actual Schema file
+*should* provide all the information you need to process these events.

schemas/fcmaterials_journal-v1.0.json

@@ -0,0 +1,110 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/fcmaterials_journal/1",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event", "MarketID", "CarrierName", "CarrierID", "Items" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "FCMaterials" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Boolean value copied from the Journal LoadGame event, when it is present there."
+                },
+                "odyssey": {
+                    "type"          : "boolean",
+                    "description"   : "Boolean value copied from the Journal LoadGame event, when it is present there."
+                },
+
+                "MarketID": {
+                    "type"          : "integer"
+                },
+
+                "CarrierName": {
+                    "type"          : "string",
+                    "minLength"     : 1
+                },
+
+                "CarrierID": {
+                    "type"          : "string",
+                    "minLength"     : 1
+                },
+
+                "Items": {
+                    "type"          : "array",
+                    "items": {
+                        "type"          : "object",
+                        "required"      : [ "id", "Name", "Price", "Stock", "Demand" ],
+                        "properties"    : {
+                            "id" : {
+                                "type"  : "integer"
+                            },
+                            "Name": {
+                                "type"      : "string",
+                                "minLength" : 1
+                            },
+                            "Price": {
+                                "type"      : "integer"
+                            },
+                            "Stock": {
+                                "type"      : "integer"
+                            },
+                            "Demand": {
+                                "type"      : "integer"
+                            }
+                        },
+                        "patternProperties": {
+                            "_Localised$"       : { "$ref" : "#/definitions/disallowed" }
+                        }
+                    }
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/fssallbodiesfound-README.md

@@ -0,0 +1,36 @@
+# EDDN FSSAllBodiesFound Schema
+
+## Introduction
+Here we document how to take data from an ED `FSSAllBodiesFound` Journal 
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The primary data source for this schema is the ED Journal event 
+`FSSAllBodiesFound`.
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
+#### StarPos
+You MUST add a `StarPos` array containing the system co-ordinates from the 
+last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**

schemas/fssallbodiesfound-v1.0.json

@@ -0,0 +1,85 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/fssallbodiesfound/1#",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal, minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event", "SystemName", "StarPos", "SystemAddress", "Count" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "FSSAllBodiesFound" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Whether the sending Cmdr has a Horizons pass."
+                },
+                "odyssey": {
+                    "type"      : "boolean",
+                    "description" : "Whether the sending Cmdr has an Odyssey expansion."
+                },
+                "SystemName": {
+                    "type"          : "string",
+                    "minLength"     : 1
+                },
+                "StarPos": {
+                    "type"          : "array",
+                    "items"         : { "type": "number" },
+                    "minItems"      : 3,
+                    "maxItems"      : 3,
+                    "description"   : "Must be added by the sender if not present in the journal event"
+                },
+                "SystemAddress": {
+                    "type"          : "integer"
+                },
+                "Count"             : {
+                    "type"          : "integer",
+                    "description"   : "Number of bodies in this system"
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/fssbodysignals-README.md

@@ -0,0 +1,53 @@
+# EDDN FSSBodySignals Schema
+
+## Introduction
+Here we document how to take data from an ED `FSSBodySignals` Journal 
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The primary data source for this schema is the ED Journal event 
+`FSSBodySignals`.
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
+#### StarSystem
+You MUST add a `StarSystem` string containing the name of the system from the 
+last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
+#### StarPos
+You MUST add a `StarPos` array containing the system co-ordinates from the 
+last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
+#### Remove _Localised key/values
+All keys whose name ends with `_Localised`, i.e. the `Type_Localised`
+key/values in Signals.
+
+#### Examples:
+
+```json
+{ "timestamp":"2022-05-18T00:10:57Z", "event":"FSSBodySignals", "BodyName":"Phoi Auwsy ZY-Z d132 7 a", "BodyID":37, "SystemAddress":4546986398603, "Signals":[ { "Type":"$SAA_SignalType_Geological;", "Type_Localised":"Geological", "Count":2 } ] }
+```

schemas/fssbodysignals-v1.0.json

@@ -0,0 +1,104 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/fssbodysignals/1#",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal, minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event", "StarSystem", "StarPos", "SystemAddress", "BodyID", "Signals" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "FSSBodySignals" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Whether the sending Cmdr has a Horizons pass."
+                },
+                "odyssey": {
+                    "type"      : "boolean",
+                    "description" : "Whether the sending Cmdr has an Odyssey expansion."
+                },
+                "StarSystem": {
+                    "type"          : "string",
+                    "minLength"     : 1
+                },
+                "StarPos": {
+                    "type"          : "array",
+                    "items"         : { "type": "number" },
+                    "minItems"      : 3,
+                    "maxItems"      : 3,
+                    "description"   : "Must be added by the sender if not present in the journal event"
+                },
+                "SystemAddress": {
+                    "type"          : "integer"
+                },
+                "BodyID"             : {
+                    "type"          : "integer"
+                },
+                "BodyName": {
+                    "type"          : "string",
+                    "minLength"     : 1
+                },
+                "Signals": {
+                    "type"          : "array",
+                    "items"         : {
+                        "type"          : "object",
+                        "additionalProperties"  : false,
+                        "required"              : [ "Type", "Count" ],
+                        "properties"            : {
+                            "Type"             : {
+                                "type"          : "string"
+                            },
+                            "Count"             : {
+                                "type"          : "integer"
+                            }
+                        }
+                    }
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/fssdiscoveryscan-README.md

@@ -7,26 +7,30 @@ Event and properly structure it for sending to EDDN.
 Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
 documentation for a schema such as this.
 
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
 ## Senders
 The primary data source for this schema is the ED Journal event 
 `FSSDiscoveryScan`.
 
-### Key Renames
-Many of the key names have a different case defined in this schema, make 
-sure you are renaming them as appropriate.
-
-### Elisions
-You MUST remove the following key/value pairs from the data:
-
-  - `Progress` key/value pair.
-
 ### Augmentations
-#### horizons flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
 
-#### odyssey flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
 
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the 
-last `FSDJump`, `CarrierJump`, or `Location` event.
+last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**

schemas/fssdiscoveryscan-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },
@@ -62,8 +70,7 @@
                     "description"   : "Must be added by the sender if not present in the journal event"
                 },
                 "SystemAddress": {
-                    "type"          : "integer",
-                    "description"   : "Should be added by the sender if not present in the journal event"
+                    "type"          : "integer"
                 },
                 "Progress"            : {
                     "$ref" : "#/definitions/disallowed",

schemas/fsssignaldiscovered-README.md

@@ -0,0 +1,191 @@
+# EDDN FSSSignalDiscovered Schema
+
+## Introduction
+Here we document how to take data from an ED `FSSSignalDiscovered` Journal 
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+## Senders
+The only data source for this schema is the ED Journal event 
+`FSSSignalDiscovered`.
+
+### Batching
+You MUST coalesce contiguous runs of `FSSSignalDiscovered` events into a
+single `signals` array in the message. Minimum size of `signals` is 1 item.
+
+Do not make a request for every single event other than where they occur
+singly (such as when a player utilises the FSS to zoom into USS individually,
+if there is a different following event).
+
+Suggested algorithm for batching:
+
+1. You will need to track the current location from `Location`, `FSDJump` and
+  `CarrierJump` events.  This is in order to add the top-level augmentation
+  of `StarSystem` (system name) and `StarPos`.  You will need to record:
+    1. `SystemAddress` - for cross-checking.
+    2. `StarSystem` - name of the star system.
+    3. `StarPos` - the galactic co-ordinates of the system.
+2. If the event is `FSSSignalDiscovered`, store it to the temporal list.
+3. If the event is any other, then:
+    1. check if it is `Location`, `FSDJump` or `CarrierJump` - if so you should
+      use this new location in the message for the augmentations.
+    2. If it is not one of those events then you should use the tracked
+      location from the prior such event for the augmentations.
+   
+    Now construct the full `fsssignaldiscovered` schema message using the
+    tracked location and the stored list of events.  *You **MUST** check that
+    the `SystemAddress` for each `FSSSignalDiscovered` event matches the
+    tracked location.*  If there is a mis-match then drop that event.
+4. Use the `timestamp` of the first signal in the batch as the top-level
+   `timestamp` in the `message` object.
+
+Point 3i/ii above are because in the current (3.8.0.406) Horizons client the
+`FSSSignalDiscovered` events arrive after `Location`/`FSDJump`/`CarrierJump`,
+but in the current (4.0.0.1302) Odyssey client they arrive before such events.
+
+Thus, in Horizons you use the last-tracked location, but in Odyssey you use
+the "just arrived" location.
+
+Manually FSS-scanned USS type signals will come in one by one, possibly with
+other events between them (such as `Music` due to zooming in/out in FSS).
+There is no need to attempt batching these together if separated by other
+events, even though you'll be using the `timestamp` of the first on the
+message, despite the actual time-line being dependent on how quickly the
+player scans them.
+
+This batching is more concerned with not causing an EDDN message per event
+upon entry into a system.
+
+### Elisions
+Remove the `event` key/pair from each member of the `signals` array.  Including
+this would be redundant as by definition we're sending `FSSSignalDiscovered`
+events on this schema.
+
+You MUST remove the following key/value pairs from the data:
+
+  - `TimeRemaining` key/value pair (will be present on USS).  This has a slight
+    PII nature and is also very ephemeral.
+
+You MUST drop the whole `FSSSignalDiscovered` event if `USSType` key
+has `$USS_Type_MissionTarget;` value.  Only the Cmdr with the mission has any
+use of these.  There's not even a statistical use.
+
+Because of the location cross-check the `SystemAddress` is in the top-level
+`message` object, and thus you **MUST** remove such from each signal in the
+array.
+
+Do **NOT** remove the `timestamp` from each signal in the array.  Whilst these
+should be identical for a "just logged in or arrived in system" set of signals,
+this is not true of manually FSS scanned USS signals.
+
+### Augmentations
+#### horizons flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### odyssey flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
+#### StarSystem
+You **MUST** add a `StarSystem` string containing the system name from the last
+tracked location.  You **MUST** cross-check each `FSSSignalDiscovered`
+->`SystemAddress` value to ensure it matches.  If it does not, you **MUST**
+drop the event.
+
+#### StarPos
+You **MUST** add a `StarPos` array containing the system co-ordinates from the
+tracked location.  You **MUST** cross-check each `FSSSignalDiscovered`
+->`SystemAddress` value to ensure it matches.  If it does not, you **MUST**
+drop the event.
+
+## Receivers
+### Augmentations are 'SHOULD', not 'MUST'
+Receivers should remember that  `horizons` and  `odyssey` augmentations
+are optional key/value pairs.  You **SHOULD NOT** rely on them being present
+in any given event.
+
+### Duplicate messages from 'busy' systems
+When a system is particularly full of signals, such as when many Fleet Carriers
+are present, it has been observed that the game repeats the identical
+sequence of `FSSSignalDiscovered` events.  So you might receive what looks like
+a duplicate message, other than the timestamp (if the timestamp is the same
+then the EDDN Relay should drop the duplicate).
+
+## Examples
+This is a few example of messages that passes current `FSSSignalDiscovered` schema.
+1. A message without `horizons` or `odyssey` augmentations.
+```json
+{
+   "$schemaRef":"https://eddn.edcd.io/schemas/fsssignaldiscovered/1",
+   "header":{
+      "gatewayTimestamp":"2021-11-06T22:48:43.483147Z",
+      "softwareName":"a software",
+      "softwareVersion":"a version",
+      "uploaderID":"an uploader"
+   },
+   "message":{
+      "timestamp":"2021-11-06T22:48:42Z",
+      "event":"FSSSignalDiscovered",
+      "SystemAddress":1774711381,
+      "signals":[
+         {
+            "timestamp":"2021-11-06T22:48:42Z",
+            "SignalName":"EXPLORER-CLASS X2X-74M",
+            "IsStation":true
+         }
+      ],
+      "StarSystem":"HR 1185",
+      "StarPos": [
+          -64.66, -148.94, -330.41
+      ]
+   }
+}
+```
+
+2. A message with `horizons`, `odyssey`, `systemName`, `StarPos` fields which says it sent from Odyssey.
+```json
+{
+   "$schemaRef":"https://eddn.edcd.io/schemas/fsssignaldiscovered/1",
+   "header":{
+      "gatewayTimestamp":"2021-11-06T22:48:43.483147Z",
+      "softwareName":"a software",
+      "softwareVersion":"a version",
+      "uploaderID":"an uploader"
+   },
+   "message":{
+      "timestamp":"2021-11-06T22:48:42Z",
+      "event":"FSSSignalDiscovered",
+      "SystemAddress":1350507186531,
+      "signals":[
+         {
+            "timestamp":"2021-11-06T22:48:42Z",
+            "event":"FSSSignalDiscovered",
+            "SignalName":"EXPLORER-CLASS X2X-74M",
+            "IsStation":true
+         },
+         {
+            "timestamp":"2021-11-06T22:48:42Z",
+            "event":"FSSSignalDiscovered", 
+            "SignalName":"$USS_NonHumanSignalSource;", 
+            "USSType":"$USS_Type_NonHuman;",
+            "SpawningState":"$FactionState_None;", 
+            "SpawningFaction":"$faction_none;",
+            "ThreatLevel":5
+            }
+      ],
+      "StarPos": [
+          8.1875,
+          124.21875,
+          -38.5
+      ],
+      "StarSystem": "HIP 56186",
+      "horizons": true,
+      "odyssey": true
+   }
+}
+```

schemas/fsssignaldiscovered-v1.0.json

@@ -0,0 +1,120 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/fsssignaldiscovered/1#",
+    "description"           : "EDDN schema for FSSSignalDiscovered Journal events.  Full documentation at https://github.com/EDCD/EDDN/tree/master/schemas/fsssignaldiscovered-README.md",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal minus Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "event", "timestamp", "SystemAddress", "StarSystem", "StarPos", "signals"],
+            "properties"            : {
+                "event": {
+                    "enum"          : [ "FSSSignalDiscovered" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Whether the sending Cmdr has a Horizons pass."
+                },
+                "odyssey": {
+                    "type"      : "boolean",
+                    "description" : "Whether the sending Cmdr has an Odyssey expansion."
+                },
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Duplicate of the first signal's timestamp, for commonality with other schemas."
+                },
+                "SystemAddress": {
+                    "type": "integer"
+                },
+                "signals": {
+                    "type": "array",
+                    "description": "Array of FSSSignalDiscovered events",
+                    "minItems": 1,
+                    "items": {
+                        "type": "object",
+                        "additionalProperties"  : false,
+                        "required": ["timestamp", "SignalName"],
+                        "description": "Single FSSSignalDiscovered event",
+                        "properties": {
+                            "timestamp": {
+                                "type"          : "string",
+                                "format"        : "date-time"
+                            },
+                            "SignalName": { "type":  "string" },
+                            "SignalType": { "type":  "string" },
+                            "IsStation": { "type": "boolean" },
+                            "USSType": {
+                              "type": "string",
+                               "not": {
+                                    "pattern": "^\\$USS_Type_MissionTarget;$"
+                                }
+                            },
+                            "TimeRemaining": {"$ref" : "#/definitions/disallowed"},
+                            "SpawningState": {"type":  "string"},
+                            "SpawningFaction" : {"type":  "string"},
+                            "SpawningPower" : {"type":  "string"},
+                            "OpposingPower" : {"type":  "string"},
+                            "ThreatLevel": {"type": "integer" },
+
+                            "patternProperties": {
+                                "_Localised$"       : { "$ref" : "#/definitions/disallowed" }
+                            }
+                        }
+                    }
+                },
+                "StarSystem": {
+                    "type"          : "string",
+                    "minLength"     : 1,
+                    "description": "Should be added by the sender"
+                },
+                "StarPos": {
+                    "type"          : "array",
+                    "items"         : { "type": "number" },
+                    "minItems"      : 3,
+                    "maxItems"      : 3,
+                    "description"   : "Should be added by the sender"
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/journal-README.md

@@ -65,22 +65,35 @@ The following keys+values should be removed from `Location` event data:
 - `SquadronFaction` from within the list of `Factions`.
 
 ### Augmentations
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### horizons flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+You **MUST** add this key/value pair, using the value from the `LoadGame` event.
+
+Note caveats in [docs/Developers.md](../docs/Developers.md).
 
 #### odyssey flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+You **MUST** add this key/value pair, using the value from the `LoadGame` event.
+
+Note caveats in [docs/Developers.md](../docs/Developers.md).
 
 #### StarSystem
-You MUST add a `StarSystem` key/value pair representing the name of the 
-system this event occurred in.  Source this from either `Location`, 
-`FSDJump` or `CarrierJump` as appropriate.
+If not already present, you MUST add a `StarSystem` string containing the
+name of the system from the last `FSDJump`, `CarrierJump`, or `Location` event.
 
-#### SystemAddress
-You MUST add a `SystemAddress` key/value pair representing the numerical ID 
-of the system this event occurred in.  Source this from either `Location`,
-`FSDJump` or `CarrierJump` as appropriate.
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
+This should only apply to `SAASignalsFound` events.
 
 #### StarPos
-You MUST add a `StarPos` array containing the system co-ordinates from the
-last `FSDJump`, `CarrierJump`, or `Location` event.
+If not already present, you  MUST add a `StarPos` array containing the
+system co-ordinates from the last `FSDJump`, `CarrierJump`, or `Location`
+event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
+This should only apply to `Docked`, `Scan` and `SAASignalsFound` events.

schemas/journal-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/navbeaconscan-README.md

@@ -7,25 +7,38 @@ Event and properly structure it for sending to EDDN.
 Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
 documentation for a schema such as this.
 
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
 ## Senders
 The primary data source for this schema is the ED Journal event
 `NavBeaconScan`.
 
-### Elisions
-There are no elisions in this schema.
-
 ### Augmentations
-#### horizons flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
 
-#### odyssey flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
 
 #### StarSystem
 You MUST add a `StarSystem` key/value pair representing the name of the
 system this event occurred in.  Source this from either `Location`,
 `FSDJump` or `CarrierJump` as appropriate.
 
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the
 last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**

schemas/navbeaconscan-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },
@@ -52,7 +60,8 @@
                 },
                 "StarSystem": {
                     "type"          : "string",
-                    "minLength"     : 1
+                    "minLength"     : 1,
+                    "description"   : "Should be added by the sender if not present in the journal event"
                 },
                 "StarPos": {
                     "type"          : "array",
@@ -62,8 +71,7 @@
                     "description"   : "Must be added by the sender if not present in the journal event"
                 },
                 "SystemAddress": {
-                    "type"          : "integer",
-                    "description"   : "Should be added by the sender if not present in the journal event"
+                    "type"          : "integer"
                 },
                 "NumBodies"         : {
                     "type"          : "integer"

schemas/navroute-README.md

@@ -7,6 +7,14 @@ Event and properly structure it for sending to EDDN.
 Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
 documentation for a schema such as this.
 
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
 ## Senders
 The primary data source for this schema is the `NavRoute.json` file.  That 
 it has been freshly written is signalled by the ED Journal event `NavRoute`.
@@ -18,12 +26,12 @@ data you got from reading this file, not merely the Journal event.
 The primary data to be sent is the `Route` array from the contents of the 
 separate file.
 
-### Elisions
-There are no elisions in this schema.
-
 ### Augmentations
-#### horizons flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
 
-#### odyssey flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.

schemas/navroute-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/outfitting-README.md

@@ -7,20 +7,35 @@ Event and properly structure it for sending to EDDN.
 Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
 documentation for a schema such as this.
 
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
 ## Senders
 The primary data source for this schema is the ED Journal event
 `Outfitting`.
 
 You MAY also source this data from the CAPI `/shipyard` endpoint.
 Please read
-[the guidance on checking for CAPI lag](README-EDDN-schemas.md#detecting-capi-data-lag)
+[the guidance on checking for CAPI lag](../docs/Developers.md#detecting-capi-data-lag)
 before utilising CAPI data for EDDN messages.
 
 You only need the `name` key's value for each member of the `modules` array.
 
 ### Key Renames
-Many of the key names have a different case defined in this schema, make
-sure you are renaming them as appropriate.
+Some key names in this Schema are different from how they appear in the source
+Journal data.  Look for keys where the object contains a `renamed` key - the
+value is what the name would have been in the source Journal data.
+
+### The modules/Items list
+The source data, Journal or CAPI, contains more than just the names of the
+available items.  This Schema is only concerned with the names, so the list
+you build will have only strings as its members, not including other information
+such as id, category, cost/BuyPrice, sku or stock.
 
 ### Elisions
 Remove items whose availability depends on the Cmdr's status rather than on the
@@ -35,8 +50,11 @@ station. Namely:
 - The `"Int_PlanetApproachSuite"` module (for historical reasons).
 
 ### Augmentations
-#### horizons flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
 
-#### odyssey flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.

schemas/outfitting-v2.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->gameversion, else LoadGame->gameversion, else 'CAPI-shipyard', else ''."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->build, else LoadGame->build, else 'CAPI-shipyard', else ''."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },
@@ -36,14 +44,17 @@
             "properties"            : {
                 "systemName": {
                     "type"      : "string",
+                    "renamed"   : "StarSystem",
                     "minLength" : 1
                 },
                 "stationName": {
                     "type"      : "string",
+                    "renamed"   : "StationName",
                     "minLength" : 1
                 },                
                 "marketId": {
-                    "type"      : "integer"
+                    "type"      : "integer",
+                    "renamed"   : "MarketID"
                 },
                 "horizons": {
                     "type"      : "boolean",
@@ -59,6 +70,7 @@
                 },
                 "modules": {
                     "type"          : "array",
+                    "renamed"       : "Items",
                     "minItems"      : 1,
                     "uniqueItems"   : true,
                     "items"         : {

schemas/scanbarycentre-README.md

@@ -7,20 +7,33 @@ Event and properly structure it for sending to EDDN.
 Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
 documentation for a schema such as this.
 
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
 ## Senders
 The primary data source for this schema is the ED Journal event
 `ScanBaryCentre`.
 
-### Elisions
-There are no elisions in this schema.
+Although most of the event-specific data is not specified as `required`,
+senders SHOULD include any defined in the schema if it's in the source data.
 
 ### Augmentations
-#### horizons flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
 
-#### odyssey flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
 
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the
 last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**

schemas/scanbarycentre-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },
@@ -62,8 +70,7 @@
                     "description"   : "Must be added by the sender if not present in the journal event"
                 },
                 "SystemAddress": {
-                    "type"          : "integer",
-                    "description"   : "Should be added by the sender if not present in the journal event"
+                    "type"          : "integer"
                 },
                 "BodyID": {
                     "type"          : "integer"

schemas/shipyard-README.md

@@ -7,34 +7,41 @@ Event and properly structure it for sending to EDDN.
 Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
 documentation for a schema such as this.
 
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
 ## Senders
 The primary data source for this schema is the ED Journal event
 `Shipyard`.
 
 You MAY also source this data from the CAPI `/shipyard` endpoint.
 Please read
-[the guidance on checking for CAPI lag](README-EDDN-schemas.md#detecting-capi-data-lag)
+[the guidance on checking for CAPI lag](../docs/Developers.md#detecting-capi-data-lag)
 before utilising CAPI data for EDDN messages.
 
- You only need the `name` key's value for each member of the `PriceList` 
-array (if using Journal, it will be from the `ships` array if using CAPI 
-data).
+The `ships` array is built from *only* the `name` values of either the Journal
+`PriceList` array in the `Shipyard.json` file, or from the `ships` array of
+CAPI `/shipyard` data.
 
 When using CAPI data *include* ships listed in the `"unavailable_list"` 
 property (i.e. available at this station, but not to this Cmdr).
 
-This list of ship names will go in the `ships` array in the EDDN message.
-
 ### Key Renames
-Many of the key names have a different case defined in this schema, make
-sure you are renaming them as appropriate.
-
-### Elisions
-There are no elisions in this schema.
+Some key names in this Schema are different from how they appear in the source
+Journal data.  Look for keys where the object contains a `renamed` key - the
+value is what the name would have been in the source Journal data.
 
 ### Augmentations
-#### horizons flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
 
-#### odyssey flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.

schemas/shipyard-v2.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->gameversion, else LoadGame->gameversion, else 'CAPI-shipyard', else ''."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->build, else LoadGame->build, else 'CAPI-shipyard', else ''."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },
@@ -36,14 +44,17 @@
             "properties"            : {
                 "systemName": {
                     "type"          : "string",
+                    "renamed"       : "StarSystem",
                     "minLength"     : 1
                 },
                 "stationName": {
                     "type"          : "string",
+                    "renamed"       : "StationName",
                     "minLength"     : 1
                 },                
                 "marketId": {
-                    "type"          : "integer"
+                    "type"          : "integer",
+                    "renamed"       : "MarketID"
                 },
                 "horizons": {
                     "type"          : "boolean",
@@ -63,6 +74,7 @@
                 },
                 "ships": {
                     "type"          : "array",
+                    "renamed"       : "PriceList",
                     "minItems"      : 1,
                     "uniqueItems"   : true,
                     "items"         : {

scripts/apache-log-rate

@@ -0,0 +1,119 @@
+#!/usr/bin/env python3
+# vim: wrapmargin=0 textwidth=0 smarttab expandtab tabstop=2 shiftwidth=2
+"""Process Apache access.log lines to find highest rate of /upload/."""
+
+import argparse
+from collections import deque
+import datetime
+import dateutil.parser
+import fileinput
+import re
+
+
+def process_log_file(
+  input_file: str ='-',
+  request_text: str = '/upload/',
+  window_size: int = 1,
+) -> None:
+  """
+  Process the indicated log file to determine peak rate of interesting lines.
+
+  :param input_file: Name of input file, `-` for stdin
+  :param request_text: The text that denotes an interesting line
+  :param window_size: Time, in seconds, for the window to assess
+  """
+  print(f'With:\n\tinput_file: "{input_file}"\n\trequest_text: "{request_text}"')
+  with fileinput.FileInput(files=(input_file)) as f:
+    apache_re = re.compile(r'^(?P<host>[.:0-9a-fA-F]{3,39}) - - \[(?P<datetime>[^\]]+)\] (?P<logtext>.*' + request_text + '.*)$')
+    print(f'Apache RE:\n{apache_re}\n')
+    apache_datetime_re = re.compile(
+      r'^(?P<d>[0-9]{2})/(?P<mon>[^/]{3})/(?P<YYYY>[0-9]{4}):(?P<time>[0-9]{2}:[0-9]{2}:[0-9]{2} \+[0-9]{4})$'
+    )
+
+    window_time_delta = datetime.timedelta(seconds=window_size)
+    window_count = 0
+    last_dt = None
+    window_end_longest_count = None
+    window_dts = deque()
+    line_count = 0
+    for line in f:
+      matches = apache_re.search(line)
+      if matches:
+        line_count += 1
+        # print(f'\nMatches:\n{line}')
+        # This will be referenced so many times we want a short name
+        m = apache_datetime_re.search(matches.group('datetime'))
+        this_dt_iso8601 = f'{m.group("YYYY")}-{m.group("mon")}-{m.group("d")} {m.group("time")}'
+
+        ###############################################################
+        # This code absolutely assumes that the apache log lines are
+        # in strictly increasing time sequence order.
+        #
+        # That's not necessarily true.  It has been observed that e.g.
+        # a long line for 00:24:39 can occur in the middle of lines for
+        # 00:24:40.
+        #
+        # Hopefully this doesn't happen too much.
+        ###############################################################
+        this_dt = dateutil.parser.parse(this_dt_iso8601)
+        # print(f'Timestamp: {this_dt}')
+        window_dts.append(this_dt)
+
+        # Find the oldest entry that is still within the window:
+        oldest_of_interest = this_dt - window_time_delta
+        while window_dts[0] <= oldest_of_interest:
+          window_dts.popleft()
+
+        if len(window_dts) > window_count:
+          window_count = len(window_dts)
+          window_end_longest_count = last_dt
+          # print(f'Largest window count     : {window_count:>9} ({window_count / window_size:>9}/s)')
+
+        last_dt = this_dt
+
+        # print()
+
+      else:
+        # print(f'\nNo matches:\n{line}\n')
+        pass
+
+    print(f'With window size         : {window_size:>9}')
+    print(f'Total line matching lines: {line_count:>9}')
+    print(f'Largest window count     : {window_count:>9} ({window_count / window_size:>9}/s)')
+    print(f'Busiest window ended at: {window_end_longest_count.strftime("%d/%b/%Y:%H:%M:%S")}')
+    
+
+def main():
+  parser = argparse.ArgumentParser(
+    description='Process Apache web server access.log lines, counting the number of a specific request per a unit of time.',
+  )
+  
+  parser.add_argument(
+    '--window-length',
+    metavar='<window size in seconds>',
+    required=False,
+    default=1,
+    help='The time period in which the max rate will be.',
+  )
+
+  parser.add_argument(
+    'input_file',
+    metavar='<input file name>',
+    help='Name of an Apache access.log file.  You may use "-" for standard input.',
+  )
+
+  parser.add_argument(
+    'request_text',
+    metavar='<per-request text selector>',
+    help='Text that appears in the log lines of interest.  Defaults to "/upload/"',
+    nargs='?',
+    default='/upload/',
+  )
+
+  args = parser.parse_args()
+
+  process_log_file(input_file=args.input_file, request_text=args.request_text, window_size=int(args.window_length))
+
+
+if __name__ == '__main__':
+  main()

scripts/eddn-report-log-errors

@@ -3,8 +3,11 @@
 """Produce a report on the provided EDDN Gateway log file's ERRORs."""
 
 import argparse
+import fileinput
 import re
 
+import semantic_version
+
 
 def parse_cl_args() -> str:
   """
@@ -43,7 +46,7 @@ def process_file(input_file: str) -> None:
     r' from (?P<sender_ip>.+)$'
   )
   # TODO: Make this handle gzipped files
-  with open(input_file, 'r') as input:
+  with fileinput.FileInput(files=(input_file), mode='r') as input:
     line = input.readline()
     while line:
       line = line.strip()
@@ -59,50 +62,81 @@ def process_file(input_file: str) -> None:
         # print(matches.group('sender_ip'))
         # print('')
 
+        try:
+          software_version = semantic_version.Version.coerce(matches.group('software_version'))
+
+        except ValueError as e:
+          print(f"Error parsing sofwareVersion for:\n{matches.group('software_version')}\n{line}\n")
+          next
+
         ###################################################################
         # Issues we know about and HAVE already alerted their
         # developers to.
         ###################################################################
         if matches.group('software_name') == 'EDDiscovery':
           # https://github.com/EDDiscovery/EDDiscovery/releases/latest
-          if matches.group('software_version') == '12.1.7.0':
-            if matches.group('schema_ref') in (
-              'https://eddn.edcd.io/schemas/shipyard/2',
-              'https://eddn.edcd.io/schemas/outfitting/2',
-            ):
-              # Reported via Discord PM to Robby 2022-01-07
-              if matches.group('err_msg') != 'Failed Validation "[<ValidationError: \'[] is too short\'>]"':
+          if software_version >= semantic_version.Version.coerce('16.0.5.0'):
+            if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/outfitting/2':
+              err_msg = matches.group('err_msg')
+              if (
+                err_msg.startswith('Failed Validation "[<ValidationError: "\'paintjob_') and
+                err_msg.find('\' does not match \'(^Hpt_|^hpt_|^Int_|^int_|_Armour_|_armour_)\'">]') != -1
+              ):
+                # <https://github.com/EDDiscovery/EDDiscovery/issues/3328>
+                pass
+
+              else: 
                 print(line)
 
             else:
               print(line)
 
         elif matches.group('software_name') == 'EDDLite':
-          # https://github.com/EDDiscovery/EDDLite/releases/tag/latest
-          if matches.group('software_version') == '2.0.0':
-            if matches.group('schema_ref') in (
-              'https://eddn.edcd.io/schemas/shipyard/2',
-              'https://eddn.edcd.io/schemas/outfitting/2',
-            ):
-              # Reported via Discord PM to Robby 2022-01-07
-              if matches.group('err_msg') != 'Failed Validation "[<ValidationError: \'[] is too short\'>]"':
-                print(line)
-
-            else:
-              print(line)
+          # https://github.com/EDDiscovery/EDDLite/releases/latest
+          if software_version >= semantic_version.Version.coerce('2.5.0'):
+            print(line)
 
         elif matches.group('software_name') == 'EDDI':
           # https://github.com/EDCD/EDDI/releases/latest
-          if matches.group('software_version') == '4.0.1':
-            print(line)
+          if software_version >= semantic_version.Version.coerce('4.0.2'):
 
-        elif matches.group('software_name') == 'E:D Market Connector [Windows]':
+            if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/fsssignaldiscovered/1':
+              if matches.group('err_msg').startswith(
+                'Failed Validation "[<ValidationError: "\'StarPos\' is a required property"'
+                ):
+                # Reported on Discord: <https://discord.com/channels/164411426939600896/353595704658231299/1062652620986134608>
+                pass
+             
+            elif matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/navroute/1':
+              if matches.group('err_msg').startswith(
+                'Failed Validation "[<ValidationError: "\'Route\' is a required property"'
+                ):
+                # Reported on Discord: <https://discord.com/channels/164411426939600896/353595704658231299/1063017819752648775>
+                pass
+             
+            else:
+              print(line)
+
+        elif matches.group('software_name').startswith('E:D Market Connector'):
           # https://github.com/EDCD/EDMarketConnector/releases/latest
-          if matches.group('software_version') == '5.2.4':
-            if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/codexentry/1':
-              # <https://github.com/EDCD/EDMarketConnector/issues/1393>
-              if matches.group('err_msg') != 'Failed Validation "[<ValidationError: "\'\' is too short">]"':
-                print(matches.group('err_msg'))
+          if software_version >= semantic_version.Version.coerce('5.7.0'):
+            if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/journal/1':
+              if matches.group('err_msg').startswith(
+                'Failed Validation "[<ValidationError: "{\'type\': [\'array\', \'boolean\', \'integer\', \'number\', \'null\', \'object\', \'string\']} is not allowed for'
+                ):
+                # <https://github.com/EDCD/EDMarketConnector/issues/1403>
+                pass
+
+            elif matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/fsssignaldiscovered/1':
+              if matches.group('err_msg') == 'Failed Validation "[<ValidationError: \'[] is too short\'>]"':
+                # <https://github.com/EDCD/EDMarketConnector/issues/1598>
+                pass
+
+              elif matches.group('err_msg') == 'Failed Validation "[<ValidationError: "None is not of type \'string\'">]"':
+                # <https://github.com/EDCD/EDMarketConnector/issues/1599>
+                pass
+
+              else:
                 print(line)
 
             else:
@@ -110,25 +144,37 @@ def process_file(input_file: str) -> None:
 
         elif matches.group('software_name') == 'Elite G19s Companion App':
           # <https://edcodex.info/?m=tools&entry=212>
-          if matches.group('software_version') == '3.7.7888.21039':
+          if software_version >= semantic_version.Version.coerce('3.7.7888.21039'):
             if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/commodity/3':
+              if matches.group('err_msg') == 'Failed Validation "[<ValidationError: "Additional properties are not allowed (\'Proportion\', \'Name\' were unexpected)">]"':
               # Reported via Frontier forums: <https://forums.frontier.co.uk/threads/elite-g19s-companion-app-with-simulated-space-traffic-control.226782/post-9690204>
-              if matches.group('err_msg') != 'Failed Validation "[<ValidationError: "Additional properties are not allowed (\'Proportion\', \'Name\' were unexpected)">]"':
-                print(matches.group('err_msg'))
-                print(line)
+                pass
 
-          else:
-            print(line)
+              else:
+                print(line)
+            
+            else:
+              print(line)
 
         elif matches.group('software_name') == 'EDSM':
           # It's in-browser, no public source/releases
-          if matches.group('software_version') == '1.0.1':
+          if software_version >= semantic_version.Version.coerce('1.0.3'):
             if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/journal/1':
               if matches.group('journal_event') == 'Scan':
-                # <https://github.com/EDSM-NET/FrontEnd/issues/466>
-                if not matches.group('err_msg').startswith(
-                    'Failed Validation "[<ValidationError: "{\'type\': [\'array\', \'boolean\', \'integer\', \'number\', \'null\', \'object\', \'string\']} is not allowed for '
+                # <https://github.com/EDSM-NET/FrontEnd/issues/472>
+                if matches.group('err_msg').startswith(
+                    'Failed Validation "[<ValidationError: "None is not of type \'integer\'">]"'
                   ):
+                  pass
+
+                elif (
+                    matches.group('err_msg').startswith('Failed Validation "[<ValidationError: "{') and 
+                    matches.group('err_msg').endswith('} is not of type \'array\'">]"')
+                ):
+                  # <https://github.com/EDSM-NET/FrontEnd/issues/473>
+                  pass
+
+                else:
                   print(matches.group('err_msg'))
                   print(line)
 
@@ -140,7 +186,7 @@ def process_file(input_file: str) -> None:
 
         elif matches.group('software_name') == 'EDSM - Console':
           # It's in-browser, no public source/releases
-          if matches.group('software_version') == '1.0':
+          if software_version >= semantic_version.Version.coerce('1.0.2'):
             if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/journal/1':
               if matches.group('journal_event') == 'Scan':
                 # <https://github.com/EDSM-NET/FrontEnd/issues/466>
@@ -158,13 +204,49 @@ def process_file(input_file: str) -> None:
 
         elif matches.group('software_name') == 'EDAOS':
           # Apparently a Barry Carylon project, but no home page ?
-          if matches.group('software_version') == '1.2.3':
+          if software_version >= semantic_version.Version.coerce('1.2.3'):
             if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/journal/1':
               if matches.group('journal_event') == 'Docked':
                 # <https://discord.com/channels/164411426939600896/205369618284544000/929102478954340372>
-                if not matches.group('err_msg').startswith(
+                if matches.group('err_msg').startswith(
                     'Failed Validation "[<ValidationError: "{\'type\': [\'array\', \'boolean\', \'integer\', \'number\', \'null\', \'object\', \'string\']} is not allowed for '
                   ):
+                  pass
+
+                print(matches.group('err_msg'))
+                print(line)
+
+              else:
+                print(line)
+
+            elif matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/shipyard/2':
+              # <https://discord.com/channels/164411426939600896/205369618284544000/955030485791285258>
+              if matches.group('err_msg').startswith(
+                'Failed Validation "[<ValidationError: \'[] is too short\'>]"'
+              ):
+                pass
+
+              else:
+                print(line)
+
+            else:
+              print(line)
+
+        elif matches.group('software_name') == 'EliteLogAgent':
+          # <https://github.com/DarkWanderer/Elite-Log-Agent>
+          if software_version >= semantic_version.Version.coerce('2.0.0.660'):
+            print(line)
+
+        # <https://edcodex.info/?m=tools&entry=440>
+        # <https://bitbucket.org/JuustoKakku/moonlight/src/master/>
+        elif matches.group('software_name') == 'Moonlight':
+          if matches.group('software_version') == '1.3.4':
+            if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/journal/1':
+              if matches.group('journal_event') == 'Scan':
+                # Ref: <https://bitbucket.org/JuustoKakku/moonlight/issues/5/bad-scan-events-being-sent-to-eddn>
+                if not matches.group('err_msg').startswith(
+                    'Failed Validation "[<ValidationError: "{\'type\': [\'array\', \'boolean\', \'integer\', \'number\', \'null\', \'object\', \'string\']} is not allowed for \''
+                  ):
                   print(matches.group('err_msg'))
                   print(line)
 
@@ -174,6 +256,12 @@ def process_file(input_file: str) -> None:
             else:
               print(line)
 
+        # Abandoned/unmaintained project
+        # <https://forums.frontier.co.uk/threads/release-eva-elite-virtual-assistant-for-iphone-ipad-no-longer-working-jan-2020.245900/page-18>
+        # <https://apps.apple.com/gb/app/eva/id1098763533>
+        elif matches.group('software_name') in ('EVA [iPhone]', 'EVA [iPad]', 'EVA [Android]'):
+          pass
+
         ###################################################################
         # Issues we know about, but haven't yet alerted developers to
         ###################################################################

scripts/test-schema.py

@@ -0,0 +1,17 @@
+import sys
+
+import simplejson
+import jsonschema
+
+schema_filename = sys.argv[1]
+message_filename = sys.argv[2]
+
+schema_file = open(schema_filename, 'r')
+schema_data = schema_file.read()
+schema = simplejson.loads(schema_data)
+
+message_file = open(message_filename, 'r')
+message_data = message_file.read()
+message = simplejson.loads(message_data)
+
+jsonschema.validate(message, schema, format_checker=jsonschema.FormatChecker())

scripts/testing/gateway-responses/approachsettlement-ancient_ruins.json

@@ -0,0 +1,22 @@
+{
+  "$schemaRef": "https://eddn.edcd.io/schemas/approachsettlement/1",
+  "header": {
+    "uploaderID": "from Athanasius Testing",
+    "softwareName": "Athanasius Testing script",
+    "softwareVersion": "v0.0.1"
+  },
+  "message": {
+    "timestamp": "2022-02-18T15:02:04Z",
+    "event": "ApproachSettlement",
+    "Name": "$Ancient:#index=1;",
+    "SystemAddress": 3515254557027,
+	"StarSystem": "Synuefe XR-H d11-102",
+    "BodyID": 13,
+    "BodyName": "Synuefe XR-H d11-102 1 b",
+    "Latitude": -46.576923,
+    "Longitude": 133.985107,
+    "StarPos": [
+         357.34375, -49.34375, -74.75
+    ]
+  }
+}

scripts/testing/gateway-responses/approachsettlement-visitor_beacon.json

@@ -0,0 +1,22 @@
+{
+  "$schemaRef": "https://eddn.edcd.io/schemas/approachsettlement/1",
+  "header": {
+    "uploaderID": "from Athanasius Testing",
+    "softwareName": "Athanasius Testing script",
+    "softwareVersion": "v0.0.1"
+  },
+  "message": {
+    "timestamp":"2022-02-18T14:33:35Z",
+    "event":"ApproachSettlement",
+    "Name":"Battlegroup's Disappearance",
+    "StarSystem": "Alioth",
+    "BodyID":8,
+    "BodyName":"Alioth 1 a",
+    "Latitude":59.972752,
+    "Longitude":-84.506294,
+    "SystemAddress":1109989017963,
+    "StarPos": [
+         -33.65625, 72.46875, -20.65625
+    ]
+  }
+}

scripts/testing/gateway-responses/approachsettlement.json

@@ -0,0 +1,23 @@
+{
+  "$schemaRef": "https://eddn.edcd.io/schemas/approachsettlement/1",
+  "header": {
+    "uploaderID": "from Athanasius Testing",
+    "softwareName": "Athanasius Testing script",
+    "softwareVersion": "v0.0.1"
+  },
+  "message": {
+    "timestamp":"2021-10-14T12:37:54Z",
+    "event":"ApproachSettlement",
+    "Name":"Arnold Defence Base",
+    "MarketID":3915738368,
+    "SystemAddress":2381282543963,
+    "StarSystem": "Ix",
+    "BodyID":32,
+    "BodyName":"Ix 5 a a",
+    "Latitude":17.090912,
+    "Longitude":160.236679,
+    "StarPos": [
+         -65.21875 , 7.75 , -111.03125 
+    ]
+  }
+}

scripts/testing/gateway-responses/codexentry.json

@@ -0,0 +1,26 @@
+{
+    "$schemaRef": "https://eddn.edcd.io/schemas/codexentry/1",
+    "header": {
+        "uploaderID": "Ath Testing",
+        "softwareName": "Athanasius test code",
+        "softwareVersion": "v0.0.1"
+    },
+    "message": {
+        "timestamp":"2021-09-24T14:29:39Z",
+        "event":"CodexEntry",
+        "EntryID":1400414,
+        "Name":"$Codex_Ent_Gas_Vents_SilicateVapourGeysers_Name;",
+        "SubCategory":"$Codex_SubCategory_Geology_and_Anomalies;",
+        "Category":"$Codex_Category_Biology;",
+        "Region":"$Codex_RegionName_18;",
+        "System":"Bestia",
+        "SystemAddress":147916327267,
+        "StarPos": [
+            1.000,
+            2.000,
+            3.000
+        ],
+        "Latitude":23.197777,
+        "Longitude":51.803349
+    }
+}

scripts/testing/gateway-responses/commodity-smallest-message.json

@@ -0,0 +1 @@
+{"$schemaRef":"https://eddn.edcd.io/schemas/commodity/3","header":{"softwareName":"E:D Market Connector Windows","softwareVersion":"5.3.0-beta4extra","uploaderID":"abcdefghijklm"},"message":{"systemName":"delphi","stationName":"The Oracle","marketId":128782803,"timestamp":"2022-01-26T12:00:00Z","commodities":[]}}

scripts/testing/gateway-responses/fssallbodiesfound.json

@@ -0,0 +1,18 @@
+{
+  "$schemaRef": "https://eddn.edcd.io/schemas/fssallbodiesfound/1",
+  "message": {
+    "timestamp":"2022-01-26T16:21:00Z",
+    "event":"FSSAllBodiesFound",
+    "SystemName":"Zeta Doradus",
+    "StarPos": [
+      30.40625,-22.65625,-2.18750
+    ],
+    "SystemAddress":44853889387,
+    "Count":1
+  },
+  "header": {
+    "uploaderID": "from Athanasius Testing",
+    "softwareName": "Athanasius Testing script",
+    "softwareVersion": "v0.0.1"
+  }
+}

scripts/testing/gateway-responses/location-invalid-systemaddress-null.json

@@ -0,0 +1,154 @@
+{
+    "$schemaRef": "https://eddn.edcd.io/schemas/journal/1",
+    "header": {
+        "uploaderID": "Ath Testing",
+        "softwareName": "Athanasius test code",
+        "softwareVersion": "v0.0.1"
+    },
+    "message": {
+	"timestamp":"2022-02-03T14:40:26Z",
+        "event":"Location",
+        "DistFromStarLS":11225.915347,
+        "Docked":true,
+        "StationName":"X3F-N5Z",
+        "StationType":"FleetCarrier",
+        "MarketID":3706433792,
+        "StationFaction":{ "Name":"FleetCarrier" },
+        "StationGovernment":"$government_Carrier;",
+        "StationServices":[ "dock",
+            "autodock",
+            "blackmarket",
+            "commodities",
+            "contacts",
+            "exploration",
+            "outfitting",
+            "crewlounge",
+            "rearm",
+            "refuel",
+            "repair",
+            "shipyard",
+            "engineer",
+            "flightcontroller",
+            "stationoperations",
+            "stationMenu",
+            "carriermanagement",
+            "carrierfuel",
+            "livery",
+            "voucherredemption"
+        ],
+        "StationEconomy":"$economy_Carrier;",
+        "StationEconomies":[
+            {
+                "Name":"$economy_Carrier;",
+                "Proportion":1.000000
+            }
+        ],
+        "Taxi":false,
+        "Multicrew":false,
+        "StarSystem":"Nuenets",
+        "SystemAddress":null,
+        "StarPos":[48.00000,17.65625,28.31250],
+        "SystemAllegiance":"Federation",
+        "SystemEconomy":"$economy_Industrial;",
+        "SystemSecondEconomy":"$economy_Extraction;",
+        "Population":70078377,
+        "Body":"Nuenets C 2",
+        "BodyID":26,
+        "BodyType":"Planet",
+        "Factions":[
+            {
+                "Name":"V886 Centauri Future",
+                "FactionState":"Boom",
+                "Government":"Democracy",
+                "Influence":0.305305,
+                "Allegiance":"Federation",
+                "Happiness":"$Faction_HappinessBand2;",
+                "ActiveStates":[ { "State":"Boom" } ]
+            },
+            {
+                "Name":"Revolutionary Party of Nuenets",
+                "FactionState":"Election",
+                "Government":"Democracy",
+                "Influence":0.155155,
+                "Allegiance":"Federation",
+                "Happiness":"$Faction_HappinessBand2;",
+                "ActiveStates":[ { "State":"Election" } ]
+            },
+            {
+                "Name":"Nuenets Jet State Limited",
+                "FactionState":"Election",
+                "Government":"Corporate",
+                "Influence":0.125125,
+                "Allegiance":"Independent",
+                "Happiness":"$Faction_HappinessBand2;",
+                "ActiveStates":[ { "State":"Election" } ]
+            },
+            {
+                "Name":"Constitution Party of Nuenets",
+                "FactionState":"None",
+                "Government":"Dictatorship",
+                "Influence":0.124124,
+                "Allegiance":"Independent",
+                "Happiness":"$Faction_HappinessBand2;"
+            },
+            { "Name":"Nuenets Blue Crew",
+                "FactionState":"None",
+                "Government":"Anarchy",
+                "Influence":0.010010,
+                "Allegiance":"Independent",
+                "Happiness":"$Faction_HappinessBand2;"
+            },
+            { "Name":"Collective of Independent Agents",
+                "FactionState":"Election",
+                "Government":"Corporate",
+                "Influence":0.155155,
+                "Allegiance":"Federation",
+                "Happiness":"$Faction_HappinessBand2;",
+                "PendingStates":[ { "State":"Expansion",
+                "Trend":0 } ],
+                "ActiveStates":[ { "State":"Outbreak" },
+                { "State":"Election" } ]
+            },
+            { "Name":"Nuenets Corp.",
+                "FactionState":"Election",
+                "Government":"Corporate",
+                "Influence":0.125125,
+                "Allegiance":"Federation",
+                "Happiness":"$Faction_HappinessBand2;",
+                "ActiveStates":[ { "State":"Election" } ]
+            }
+        ],
+        "SystemFaction":{ "Name":"V886 Centauri Future",
+        "FactionState":"Boom" },
+        "Conflicts":[
+            {
+                "WarType":"election",
+                "Status":"active",
+                "Faction1":{
+                    "Name":"Revolutionary Party of Nuenets",
+                    "Stake":"Sibanda Industrial Works",
+                    "WonDays":0
+                },
+                "Faction2":{
+                    "Name":"Collective of Independent Agents",
+                    "Stake":"Harbaugh Port",
+                    "WonDays":0
+                }
+            },
+            {
+	        "WarType":"election",
+                "Status":"active",
+                "Faction1":{
+                    "Name":"Nuenets Jet State Limited",
+                    "Stake":"Bellegrade's Nook",
+                    "WonDays":0
+                },
+                "Faction2":{
+                    "Name":"Nuenets Corp.",
+                    "Stake":"Watson's Shelter",
+                    "WonDays":0
+                }
+            }
+        ]
+    }
+}

scripts/testing/gateway-responses/location.json

@@ -0,0 +1,154 @@
+{
+    "$schemaRef": "https://eddn.edcd.io/schemas/journal/1",
+    "header": {
+        "uploaderID": "Ath Testing",
+        "softwareName": "Athanasius test code",
+        "softwareVersion": "v0.0.1"
+    },
+    "message": {
+	"timestamp":"2022-02-03T14:40:26Z",
+        "event":"Location",
+        "DistFromStarLS":11225.915347,
+        "Docked":true,
+        "StationName":"X3F-N5Z",
+        "StationType":"FleetCarrier",
+        "MarketID":3706433792,
+        "StationFaction":{ "Name":"FleetCarrier" },
+        "StationGovernment":"$government_Carrier;",
+        "StationServices":[ "dock",
+            "autodock",
+            "blackmarket",
+            "commodities",
+            "contacts",
+            "exploration",
+            "outfitting",
+            "crewlounge",
+            "rearm",
+            "refuel",
+            "repair",
+            "shipyard",
+            "engineer",
+            "flightcontroller",
+            "stationoperations",
+            "stationMenu",
+            "carriermanagement",
+            "carrierfuel",
+            "livery",
+            "voucherredemption"
+        ],
+        "StationEconomy":"$economy_Carrier;",
+        "StationEconomies":[
+            {
+                "Name":"$economy_Carrier;",
+                "Proportion":1.000000
+            }
+        ],
+        "Taxi":false,
+        "Multicrew":false,
+        "StarSystem":"Nuenets",
+        "SystemAddress":3240309557611,
+        "StarPos":[48.00000,17.65625,28.31250],
+        "SystemAllegiance":"Federation",
+        "SystemEconomy":"$economy_Industrial;",
+        "SystemSecondEconomy":"$economy_Extraction;",
+        "Population":70078377,
+        "Body":"Nuenets C 2",
+        "BodyID":26,
+        "BodyType":"Planet",
+        "Factions":[
+            {
+                "Name":"V886 Centauri Future",
+                "FactionState":"Boom",
+                "Government":"Democracy",
+                "Influence":0.305305,
+                "Allegiance":"Federation",
+                "Happiness":"$Faction_HappinessBand2;",
+                "ActiveStates":[ { "State":"Boom" } ]
+            },
+            {
+                "Name":"Revolutionary Party of Nuenets",
+                "FactionState":"Election",
+                "Government":"Democracy",
+                "Influence":0.155155,
+                "Allegiance":"Federation",
+                "Happiness":"$Faction_HappinessBand2;",
+                "ActiveStates":[ { "State":"Election" } ]
+            },
+            {
+                "Name":"Nuenets Jet State Limited",
+                "FactionState":"Election",
+                "Government":"Corporate",
+                "Influence":0.125125,
+                "Allegiance":"Independent",
+                "Happiness":"$Faction_HappinessBand2;",
+                "ActiveStates":[ { "State":"Election" } ]
+            },
+            {
+                "Name":"Constitution Party of Nuenets",
+                "FactionState":"None",
+                "Government":"Dictatorship",
+                "Influence":0.124124,
+                "Allegiance":"Independent",
+                "Happiness":"$Faction_HappinessBand2;"
+            },
+            { "Name":"Nuenets Blue Crew",
+                "FactionState":"None",
+                "Government":"Anarchy",
+                "Influence":0.010010,
+                "Allegiance":"Independent",
+                "Happiness":"$Faction_HappinessBand2;"
+            },
+            { "Name":"Collective of Independent Agents",
+                "FactionState":"Election",
+                "Government":"Corporate",
+                "Influence":0.155155,
+                "Allegiance":"Federation",
+                "Happiness":"$Faction_HappinessBand2;",
+                "PendingStates":[ { "State":"Expansion",
+                "Trend":0 } ],
+                "ActiveStates":[ { "State":"Outbreak" },
+                { "State":"Election" } ]
+            },
+            { "Name":"Nuenets Corp.",
+                "FactionState":"Election",
+                "Government":"Corporate",
+                "Influence":0.125125,
+                "Allegiance":"Federation",
+                "Happiness":"$Faction_HappinessBand2;",
+                "ActiveStates":[ { "State":"Election" } ]
+            }
+        ],
+        "SystemFaction":{ "Name":"V886 Centauri Future",
+        "FactionState":"Boom" },
+        "Conflicts":[
+            {
+                "WarType":"election",
+                "Status":"active",
+                "Faction1":{
+                    "Name":"Revolutionary Party of Nuenets",
+                    "Stake":"Sibanda Industrial Works",
+                    "WonDays":0
+                },
+                "Faction2":{
+                    "Name":"Collective of Independent Agents",
+                    "Stake":"Harbaugh Port",
+                    "WonDays":0
+                }
+            },
+            {
+	        "WarType":"election",
+                "Status":"active",
+                "Faction1":{
+                    "Name":"Nuenets Jet State Limited",
+                    "Stake":"Bellegrade's Nook",
+                    "WonDays":0
+                },
+                "Faction2":{
+                    "Name":"Nuenets Corp.",
+                    "Stake":"Watson's Shelter",
+                    "WonDays":0
+                }
+            }
+        ]
+    }
+}

scripts/testing/gateway-responses/scan-valid_but-value-looks-like-formencoding.json

@@ -0,0 +1,37 @@
+{
+  "$schemaRef": "https://eddn.edcd.io/schemas/journal/1",
+  "message": {
+    "timestamp":"2021-11-05T15:46:28Z",
+    "event":"Scan",
+    "ScanType":"AutoScan",
+    "BodyName":"Elphin=wobble& Something",
+    "BodyID":1,
+    "Parents":[ {"Null":0} ],
+    "StarSystem":"Elphin=bloop; wibble",
+    "StarPos":[-30.12500,8.18750,-17.00000],
+    "SystemAddress":3932076118738,
+    "DistanceFromArrivalLS":0.000000,
+    "StarType":"K",
+    "Subclass":3,
+    "StellarMass":0.769531,
+    "Radius":587464832.000000,
+    "AbsoluteMagnitude":6.294067,
+    "Age_MY":9558,
+    "SurfaceTemperature":4796.000000,
+    "Luminosity":"V",
+    "SemiMajorAxis":1704360246658.325195,
+    "Eccentricity":0.348740,
+    "OrbitalInclination":-72.647343,
+    "Periapsis":86.347190,
+    "OrbitalPeriod":7189218699.932098,
+    "AscendingNode":0.000000,
+    "MeanAnomaly":351.262353,
+    "RotationPeriod":248957.736717,
+    "AxialTilt":-0.126915
+  },
+  "header": {
+    "uploaderID": "from Athanasius Testing",
+    "softwareName": "Athanasius Testing script",
+    "softwareVersion": "v0.0.1"
+  }
+}

scripts/testing/gateway-responses/shipyard-smallest-message.json

@@ -0,0 +1 @@
+{"$schemaRef":"https://eddn.edcd.io/schemas/shipyard/2","header":{"softwareName":"E:D Market Connector Windows","softwareVersion":"5.3.0-beta4extra","uploaderID":"abcdefghijklm"},"message":{"systemName":"delphi","stationName":"The Oracle","marketId":128782803,"timestamp":"2022-01-26T12:00:00Z","ships":[]}}

scripts/testing/gateway-responses/test-bad-gzip.py

@@ -1,38 +0,0 @@
-#!/usr/bin/env python3
-#
-# 2022-01-10:   THIS SCRIPT DOES NOT PERFORM THE INTENDED PURPOSE
-# BECAUSE IT SEEMS THAT `requests` (or underlying modules) IS TOO CLEVER
-# AND APPLIES COMPRESSION WHEN WE SET THE `Content-Encoding: gzip`
-# HEADER
-
-import json
-import requests
-import sys
-
-print('''
-DO NOT USE THIS SCRIPT, IT DOES NOT PERFORM THE INTENDED PURPOSE.
-
-USE THE `test-bad-gzip.sh` SCRIPT INSTEAD.
-
-''')
-sys.exit(-1)
-
-if len(sys.argv) != 2:
-  print('test-sender.py <filename>')
-  sys.exit(-1)
-
-with open(sys.argv[1], 'r') as f:
-  msg = f.read()
-
-  s = requests.Session()
-
-  # This apparently causes compression to actually happen
-  s.headers['Content-Encoding'] = 'gzip'
-  r = s.post(
-    'https://beta.eddn.edcd.io:4431/upload/',
-    data=msg,
-  )
-
-  print(f'Response: {r!r}')
-  print(f'Body: {r.content.decode()}')
-

scripts/testing/gateway-responses/test-bad-gzip.sh

@@ -1,6 +0,0 @@
-#!/bin/sh
-#
-# python `requests` appears to perform compression when you set the
-# 'Content-Encoding: gzip' header, so do this with curl.
-
-curl --verbose -d 'wegiuweuygtfawgep9aqe8fpq2387lfbr;iufvypq38764tpgf' -H 'Content-Encoding: gzip' 'https://beta.eddn.edcd.io:4431/upload/'

scripts/testing/gateway-responses/test-gzip-bad-formdata.py

@@ -1,37 +0,0 @@
-#!/usr/bin/env python3
-
-import json
-import requests
-import sys
-import urllib3
-import zlib
-
-if len(sys.argv) != 2:
-  print('test-sender.py <filename>')
-  sys.exit(-1)
-
-with open(sys.argv[1], 'r') as f:
-  # Read from provided file
-  msg = f.read()
-
-  # Fake form-encode it
-  msg = 'wibble=' + msg
-
-  # Compress it
-  msg_gzip = zlib.compress(msg.encode('utf-8'))
-
-  http = urllib3.PoolManager()
-
-  # Send that compressed data as a POST body
-  r = http.request(
-    'POST',
-    'https://beta.eddn.edcd.io:4431/upload/',
-    headers={
-      'Content-Encoding': 'gzip'
-    },
-    body=msg_gzip
-  )
-
-  print(f'Response: {r.status!r}')
-  print(f'Body:\n{r.data.decode()}\n')
-

scripts/testing/gateway-responses/test-gzip-correct-formdata.py

@@ -1,37 +0,0 @@
-#!/usr/bin/env python3
-
-import json
-import requests
-import sys
-import urllib3
-import zlib
-
-if len(sys.argv) != 2:
-  print('test-sender.py <filename>')
-  sys.exit(-1)
-
-with open(sys.argv[1], 'r') as f:
-  # Read from provided file
-  msg = f.read()
-
-  # Fake form-encode it
-  msg = 'data=' + msg
-
-  # Compress it
-  msg_gzip = zlib.compress(msg.encode('utf-8'))
-
-  http = urllib3.PoolManager()
-
-  # Send that compressed data as a POST body
-  r = http.request(
-    'POST',
-    'https://beta.eddn.edcd.io:4431/upload/',
-    headers={
-      'Content-Encoding': 'gzip'
-    },
-    body=msg_gzip
-  )
-
-  print(f'Response: {r.status!r}')
-  print(f'Body:\n{r.data.decode()}\n')
-

scripts/testing/gateway-responses/test-plain-bad-formdata.py

@@ -1,29 +0,0 @@
-#!/usr/bin/env python3
-
-import json
-import sys
-import urllib3
-
-if len(sys.argv) != 2:
-  print('test-sender.py <filename>')
-  sys.exit(-1)
-
-with open(sys.argv[1], 'r') as f:
-  # Read from provided file
-  msg = f.read()
-
-  # Fake form-encode it
-  msg = 'wibble=' + msg
-
-  http = urllib3.PoolManager()
-
-  # Send that data as a POST body
-  r = http.request(
-    'POST',
-    'https://beta.eddn.edcd.io:4431/upload/',
-    body=msg
-  )
-
-  print(f'Response: {r.status!r}')
-  print(f'Body:\n{r.data.decode()}\n')
-

scripts/testing/gateway-responses/test-plain-correct-formdata.py

@@ -1,29 +0,0 @@
-#!/usr/bin/env python3
-
-import json
-import sys
-import urllib3
-
-if len(sys.argv) != 2:
-  print('test-sender.py <filename>')
-  sys.exit(-1)
-
-with open(sys.argv[1], 'r') as f:
-  # Read from provided file
-  msg = f.read()
-
-  # Fake form-encode it
-  msg = 'data=' + msg
-
-  http = urllib3.PoolManager()
-
-  # Send that data as a POST body
-  r = http.request(
-    'POST',
-    'https://beta.eddn.edcd.io:4431/upload/',
-    body=msg
-  )
-
-  print(f'Response: {r.status!r}')
-  print(f'Body:\n{r.data.decode()}\n')
-

scripts/testing/gateway-responses/test-sender.py

@@ -1,20 +1,86 @@
 #!/usr/bin/env python3
+# vim: tabstop=4 shiftwidth=4 expandtab smarttab textwidth=0 wrapmargin=0
 
-import json
+import argparse
 import requests
-import sys
+import zlib
 
-if len(sys.argv) != 2:
-  print('test-sender.py <filename>')
-  sys.exit(-1)
+upload_url = 'https://dev.eddn.edcd.io:4432/upload/'
 
-with open(sys.argv[1], 'r') as f:
-  msg = f.read()
+def send_message(url, args):
+    print(f'''
+send_message:
+    URL: {url}
+    input file: "{args.messagefile}"
+''')
 
-  s = requests.Session()
+    with open(args.messagefile, 'r') as f:
+        msg = f.read()
 
-  r = s.post('https://beta.eddn.edcd.io:4431/upload/', data=msg)
+        if args.formdata:
+            if args.formdata == 'good':
+                msg = 'data=' + msg
 
-  print(f'Response: {r!r}')
-  print(f'Body: {r.content.decode()}')
+            elif args.formdata == 'bad':
+                msg = 'BADLYENCODED=' + msg
 
+        s = requests.Session()
+
+        if args.gzip:
+            # We assume that the argparse setup is enforcing the value being
+            # valid, i.e. `'good'` if it's not `'bad'`.
+            msg = zlib.compress(msg.encode('utf-8'))
+            s.headers['Content-Encoding'] = 'gzip'
+
+            if args.gzip == 'bad':
+                # Prepend a character so it's not a valid gzip header
+                msg = b'w' + msg
+
+        r = s.post(upload_url, data=msg)
+
+        print(f'Response: {r!r}')
+        print(f'Body: {r.content.decode()}')
+
+if __name__ == "__main__":
+    __parser = argparse.ArgumentParser(
+        description='Send test messages to an EDDN /upload/ endpoint',
+    )
+
+    __parser.add_argument(
+        '--url',
+        metavar='<full URL of /upload/ endpoint>',
+        help='The full URL of an EDDN /upload/ endpoint',
+    )
+
+    __parser.add_argument(
+        '--formdata',
+        choices=('good', 'bad'),
+        help='Specify to form-encode the request body',
+    )
+
+    __parser.add_argument(
+        '--gzip',
+        choices=('good', 'bad'),
+        help='Specify to gzip-compress the request body',
+    )
+
+    __parser.add_argument(
+        'messagefile',
+        metavar='<input file name>',
+        help='Name of a file containing the body of the EDDN message to be sent',
+    )
+
+    args = __parser.parse_args()
+
+    if args.url:
+        # Allow for some short aliases, but NOT!!! for live !!!
+        if args.url == 'beta':
+            upload_url = 'https://beta.eddn.edcd.io:4431/upload/'
+
+        elif args.url == 'dev':
+            upload_url = 'https://dev.eddn.edcd.io:4432/upload/'
+
+        else:
+            upload_url = args.url
+
+    send_message(upload_url, args)

setup.py

@@ -163,16 +163,6 @@ shutil.copy(
     '%s/start-eddn-%s-service' % ( START_SCRIPT_BIN, setup_env.EDDN_ENV )
 )
 
-# Ensure the service log file archiving script is in place
-print """
-******************************************************************************
-Ensuring the service log file archiving script is in place
-"""
-shutil.copy(
-    'contrib/eddn-logs-archive',
-    START_SCRIPT_BIN
-)
-
 # Ensure the latest monitor files are in place
 old_umask = os.umask(022)
 print """

src/eddn/Bouncer.py

@@ -26,6 +26,7 @@ Architecture:
   4. `push_message()` then sends the message to the configured live/real
     Gateway.
 """
+import argparse
 import gevent
 import hashlib
 import logging
@@ -47,8 +48,15 @@ from bottle import Bottle, run, request, response, get, post
 app = Bottle()
 
 logger = logging.getLogger(__name__)
-logger.setLevel(logging.DEBUG)
-logger.addHandler(logging.StreamHandler())
+logger.setLevel(logging.INFO)
+__logger_channel = logging.StreamHandler()
+__logger_formatter = logging.Formatter(
+    '%(asctime)s - %(levelname)s - %(module)s:%(lineno)d: %(message)s'
+    )
+__logger_formatter.default_time_format = '%Y-%m-%d %H:%M:%S'
+__logger_formatter.default_msec_format = '%s.%03d'
+__logger_channel.setFormatter(__logger_formatter)
+logger.addHandler(__logger_channel)
 logger.info('Made logger')
 
 
@@ -57,6 +65,27 @@ from eddn.core.StatsCollector import StatsCollector
 statsCollector = StatsCollector()
 statsCollector.start()
 
+
+def parse_cl_args():
+    parser = argparse.ArgumentParser(
+        prog='Gateway',
+        description='EDDN Gateway server',
+    )
+
+    parser.add_argument(
+        '--loglevel',
+        help='Logging level to output at',
+    )
+
+    parser.add_argument(
+        '-c', '--config',
+        metavar='config filename',
+        nargs='?',
+        default=None,
+    )
+
+    return parser.parse_args()
+
 def push_message(message_body):
     """
     Spawned as a greenlet to push messages (strings) through ZeroMQ.
@@ -243,8 +272,12 @@ class CustomLogging(object):
         return _log_to_logger
 
 def main():
+    cl_args = parse_cl_args()
+    if cl_args.loglevel:
+        logger.setLevel(cl_args.loglevel)
+
     logger.info('Loading config...')
-    loadConfig()
+    loadConfig(cl_args)
 
     logger.info('Installing EnableCors ...')
     app.install(EnableCors())

src/eddn/Gateway.py

@@ -4,11 +4,11 @@
 Contains the necessary ZeroMQ socket and a helper function to publish
 market data to the Announcer daemons.
 """
+import argparse
 import gevent
 import hashlib
 import logging
 import simplejson
-import urlparse
 import zlib
 import zmq.green as zmq
 from datetime import datetime
@@ -51,12 +51,39 @@ statsCollector = StatsCollector()
 statsCollector.start()
 
 
+def parse_cl_args():
+    parser = argparse.ArgumentParser(
+        prog='Gateway',
+        description='EDDN Gateway server',
+    )
+
+    parser.add_argument(
+        '--loglevel',
+        help='Logging level to output at',
+    )
+
+    parser.add_argument(
+        '-c', '--config',
+        metavar='config filename',
+        nargs='?',
+        default=None,
+    )
+
+    return parser.parse_args()
+
+
 def extract_message_details(parsed_message):
     uploader_id = '<<UNKNOWN>>'
     software_name = '<<UNKNOWN>>'
     software_version = '<<UNKNOWN>>'
+    game_version = '<<UNKNOWN>>'
+    game_build = '<<UNKNOWN>>'
     schema_ref = '<<UNKNOWN>>'
     journal_event = '<<UNKNOWN>>'
+    system_name = '<<UNKNOWN>>'
+    system_address = '<<UNKNOWN>>'
+    station_name = '<<UNKNOWN>>'
+    station_marketid = '<<UNKNOWN>>'
 
     if 'header' in parsed_message:
         if 'uploaderID' in parsed_message['header']:
@@ -68,6 +95,12 @@ def extract_message_details(parsed_message):
         if 'softwareVersion' in parsed_message['header']:
             software_version = parsed_message['header']['softwareVersion']
 
+        if 'gameversion' in parsed_message['header']:
+            game_version = parsed_message['header']['gameversion']
+
+        if 'gamebuild' in parsed_message['header']:
+            game_build = parsed_message['header']['gamebuild']
+
     if '$schemaRef' in parsed_message:
         schema_ref = parsed_message['$schemaRef']
 
@@ -80,7 +113,44 @@ def extract_message_details(parsed_message):
         else:
             journal_event = '-'
 
-    return uploader_id, software_name, software_version, schema_ref, journal_event
+        if 'systemName' in parsed_message['message']:
+            system_name = parsed_message['message']['systemName']
+
+        elif 'SystemName' in parsed_message['message']:
+            system_name = parsed_message['message']['SystemName']
+
+        elif 'StarSystem' in parsed_message['message']:
+            system_name = parsed_message['message']['StarSystem']
+
+        else:
+            system_name = '-'
+
+        if 'SystemAddress' in parsed_message['message']:
+            system_address = parsed_message['message']['SystemAddress']
+
+        else:
+            system_address = '-'
+
+        if 'stationName' in parsed_message['message']:
+            station_name = parsed_message['message']['stationName']
+        
+        elif 'StationName' in parsed_message['message']:
+            station_name = parsed_message['message']['StationName']
+
+        else:
+            station_name = '-'
+        
+        if 'marketId' in parsed_message['message']:
+            station_marketid = parsed_message['message']['marketId']
+
+        elif 'MarketID' in parsed_message['message']:
+            station_marketid = parsed_message['message']['MarketID']
+
+        else:
+            station_marketid = '-'
+
+    return uploader_id, software_name, software_version, schema_ref, journal_event, game_version, game_build, \
+           system_name, system_address, station_name, station_marketid
 
 def configure():
     # Get the list of transports to bind from settings. This allows us to PUB
@@ -138,56 +208,17 @@ def get_decompressed_message():
             # Auto header checking.
             logger.debug('Trying zlib.decompress (15 + 32)...')
             message_body = zlib.decompress(request.body.read(), 15 + 32)
+
         except zlib.error:
             logger.error('zlib.error, trying zlib.decompress (-15)')
             # Negative wbits suppresses adler32 checksumming.
             message_body = zlib.decompress(request.body.read(), -15)
             logger.debug('Resulting message_body:\n%s\n' % (message_body))
 
-        # At this point, we're not sure whether we're dealing with a straight
-        # un-encoded POST body, or a form-encoded POST. Attempt to parse the
-        # body. If it's not form-encoded, this will return an empty dict.
-        form_enc_parsed = urlparse.parse_qs(message_body)
-        if form_enc_parsed:
-            logger.debug('Request is form-encoded')
-            # This is a form-encoded POST. The value of the data attrib will
-            # be the body we're looking for.
-            try:
-                message_body = form_enc_parsed['data'][0]
-            except (KeyError, IndexError):
-                raise MalformedUploadError(
-                    "No 'data' POST key/value found. Check your POST key "
-                    "name for spelling, and make sure you're passing a value."
-                )
-
-        else:
-            logger.debug('Request is *NOT* form-encoded')
-
     else:
         logger.debug('Content-Encoding indicates *not* compressed...')
 
-        form_enc_parsed = urlparse.parse_qs(request.body.read())
-        if form_enc_parsed:
-            logger.debug('Request is form-encoded')
-
-            # Uncompressed request. Bottle handles all of the parsing of the
-            # POST key/vals, or un-encoded body.
-            data_key = request.forms.get('data')
-            if data_key:
-                logger.debug('form-encoded POST request detected...')
-                # This is a form-encoded POST. Support the silly people.
-                message_body = data_key
-    
-            else:
-                raise MalformedUploadError(
-                    "No 'data' POST key/value found. Check your POST key "
-                    "name for spelling, and make sure you're passing a value."
-                )
-
-        else:
-            logger.debug('Plain POST request detected...')
-            # This is a non form-encoded POST body.
-            message_body = request.body.read()
+        message_body = request.body.read()
 
     return message_body
 
@@ -236,10 +267,14 @@ def parse_and_error_handle(data):
         gevent.spawn(push_message, parsed_message, parsed_message['$schemaRef'])
 
         try:
-            uploader_id, software_name, software_version, schema_ref, journal_event = extract_message_details(parsed_message)
-            logger.info('Accepted (%d, "%s", "%s", "%s", "%s", "%s") from %s' % (
+            uploader_id, software_name, software_version, schema_ref, journal_event, game_version, game_build, \
+                system_name, system_address, station_name, station_marketid = extract_message_details(parsed_message)
+            logger.info('Accepted (%d, "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s") from %s' % (
                 request.content_length,
                 uploader_id, software_name, software_version, schema_ref, journal_event,
+                game_version, game_build,
+                system_name, system_address,
+                station_name, station_marketid,
                 get_remote_address()
             ))
 
@@ -251,11 +286,15 @@ def parse_and_error_handle(data):
 
     else:
         try:
-            uploader_id, software_name, software_version, schema_ref, journal_event = extract_message_details(parsed_message)
-            logger.error('Failed Validation "%s" (%d, "%s", "%s", "%s", "%s", "%s") from %s' % (
+            uploader_id, software_name, software_version, schema_ref, journal_event, game_version, game_build, \
+                system_name, system_address, station_name, station_marketid = extract_message_details(parsed_message)
+            logger.error('Failed Validation "%s" (%d, "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s") from %s' % (
                     str(validationResults.messages),
                     request.content_length,
                     uploader_id, software_name, software_version, schema_ref, journal_event,
+                    game_version, game_build,
+                    system_name, system_address,
+                    station_name, station_marketid,
                     get_remote_address()
             ))
 
@@ -352,7 +391,12 @@ class EnableCors(object):
 
 
 def main():
-    loadConfig()
+
+    cl_args = parse_cl_args()
+    if cl_args.loglevel:
+        logger.setLevel(cl_args.loglevel)
+
+    loadConfig(cl_args)
     configure()
 
     app.install(EnableCors())

src/eddn/Monitor.py

@@ -4,6 +4,7 @@
 Monitor sit below gateways, or another relay, and simply parse what it receives over SUB.
 """
 from threading import Thread
+import argparse
 import zlib
 import gevent
 import simplejson
@@ -27,6 +28,26 @@ if Settings.RELAY_DUPLICATE_MAX_MINUTES:
     duplicateMessages.start()
 
 
+def parse_cl_args():
+    parser = argparse.ArgumentParser(
+        prog='Gateway',
+        description='EDDN Gateway server',
+    )
+
+    parser.add_argument(
+        '--loglevel',
+        help='CURRENTLY NO EFFECT - Logging level to output at',
+    )
+
+    parser.add_argument(
+        '-c', '--config',
+        metavar='config filename',
+        nargs='?',
+        default=None,
+    )
+
+    return parser.parse_args()
+
 def date(__format):
     d = datetime.datetime.utcnow()
     return d.strftime(__format)
@@ -237,7 +258,9 @@ class EnableCors(object):
         return _enable_cors
 
 def main():
-    loadConfig()
+    cl_args = parse_cl_args()
+    loadConfig(cl_args)
+
     m = Monitor()
     m.start()
     app.install(EnableCors())

src/eddn/Relay.py

@@ -4,20 +4,32 @@
 Relays sit below an announcer, or another relay, and simply repeat what
 they receive over PUB/SUB.
 """
-# Logging has to be configured first before we do anything.
+import argparse
+import gevent
+import hashlib
 import logging
+import simplejson
+import time
+import uuid
+import zlib
 from threading import Thread
 
-import time
-
-logger = logging.getLogger(__name__)
-import zlib
-
-import gevent
-import simplejson
-import hashlib
-import uuid
 import zmq.green as zmq
+
+
+# Logging has to be configured first before we do anything.
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+__logger_channel = logging.StreamHandler()
+__logger_formatter = logging.Formatter(
+    '%(asctime)s - %(levelname)s - %(module)s:%(lineno)d: %(message)s'
+    )
+__logger_formatter.default_time_format = '%Y-%m-%d %H:%M:%S'
+__logger_formatter.default_msec_format = '%s.%03d'
+__logger_channel.setFormatter(__logger_formatter)
+logger.addHandler(__logger_channel)
+logger.info('Made logger')
+
 from eddn.conf.Settings import Settings, loadConfig
 
 from gevent import monkey
@@ -25,6 +37,7 @@ monkey.patch_all()
 from bottle import Bottle, get, request, response, run
 app = Bottle()
 
+
 # This import must be done post-monkey-patching!
 from eddn.core.StatsCollector import StatsCollector
 statsCollector = StatsCollector()
@@ -37,6 +50,26 @@ if Settings.RELAY_DUPLICATE_MAX_MINUTES:
     duplicateMessages.start()
 
 
+def parse_cl_args():
+    parser = argparse.ArgumentParser(
+        prog='Gateway',
+        description='EDDN Gateway server',
+    )
+
+    parser.add_argument(
+        '--loglevel',
+        help='Logging level to output at',
+    )
+
+    parser.add_argument(
+        '-c', '--config',
+        metavar='config filename',
+        nargs='?',
+        default=None,
+    )
+
+    return parser.parse_args()
+
 @app.route('/stats/', method=['OPTIONS', 'GET'])
 def stats():
     stats = statsCollector.getSummary()
@@ -46,7 +79,7 @@ def stats():
 
 class Relay(Thread):
 
-    REGENERATE_UPLOADER_NONCE_INTERVAL = 12 * 60 * 60  # 12 hrs
+    REGENERATE_UPLOADER_NONCE_INTERVAL = 3 * 60  # 3 minutes, was 12 hrs
 
     def __init__(self, **kwargs):
         super(Relay, self).__init__(**kwargs)
@@ -171,7 +204,12 @@ class EnableCors(object):
 
 
 def main():
-    loadConfig()
+    cl_args = parse_cl_args()
+    if cl_args.loglevel:
+        logger.setLevel(cl_args.loglevel)
+
+    loadConfig(cl_args)
+
     r = Relay()
     r.start()
 

src/eddn/conf/Settings.py

@@ -1,6 +1,5 @@
 # coding: utf8
 
-import argparse
 import simplejson
 from eddn.conf.Version import __version__ as version
 
@@ -60,20 +59,42 @@ class _Settings(object):
         "https://eddn.edcd.io/schemas/journal/1"                    : "schemas/journal-v1.0.json",
         "https://eddn.edcd.io/schemas/journal/1/test"               : "schemas/journal-v1.0.json",
 
-        "https://eddn.edcd.io/schemas/scanbarycentre/1"                    : "schemas/scanbarycentre-v1.0.json",
-        "https://eddn.edcd.io/schemas/scanbarycentre/1/test"               : "schemas/scanbarycentre-v1.0.json",
+        "https://eddn.edcd.io/schemas/scanbarycentre/1"             : "schemas/scanbarycentre-v1.0.json",
+        "https://eddn.edcd.io/schemas/scanbarycentre/1/test"        : "schemas/scanbarycentre-v1.0.json",
 
-        "https://eddn.edcd.io/schemas/fssdiscoveryscan/1"                    : "schemas/fssdiscoveryscan-v1.0.json",
-        "https://eddn.edcd.io/schemas/fssdiscoveryscan/1/test"               : "schemas/fssdiscoveryscan-v1.0.json",
+        "https://eddn.edcd.io/schemas/fssdiscoveryscan/1"           : "schemas/fssdiscoveryscan-v1.0.json",
+        "https://eddn.edcd.io/schemas/fssdiscoveryscan/1/test"      : "schemas/fssdiscoveryscan-v1.0.json",
 
-        "https://eddn.edcd.io/schemas/codexentry/1"                    : "schemas/codexentry-v1.0.json",
-        "https://eddn.edcd.io/schemas/codexentry/1/test"               : "schemas/codexentry-v1.0.json",
+        "https://eddn.edcd.io/schemas/codexentry/1"                 : "schemas/codexentry-v1.0.json",
+        "https://eddn.edcd.io/schemas/codexentry/1/test"            : "schemas/codexentry-v1.0.json",
 
-        "https://eddn.edcd.io/schemas/navbeaconscan/1"                    : "schemas/navbeaconscan-v1.0.json",
-        "https://eddn.edcd.io/schemas/navbeaconscan/1/test"               : "schemas/navbeaconscan-v1.0.json",
+        "https://eddn.edcd.io/schemas/navbeaconscan/1"              : "schemas/navbeaconscan-v1.0.json",
+        "https://eddn.edcd.io/schemas/navbeaconscan/1/test"         : "schemas/navbeaconscan-v1.0.json",
 
-        "https://eddn.edcd.io/schemas/navroute/1"                    : "schemas/navroute-v1.0.json",
-        "https://eddn.edcd.io/schemas/navroute/1/test"               : "schemas/navroute-v1.0.json",
+        "https://eddn.edcd.io/schemas/navroute/1"                   : "schemas/navroute-v1.0.json",
+        "https://eddn.edcd.io/schemas/navroute/1/test"              : "schemas/navroute-v1.0.json",
+
+        "https://eddn.edcd.io/schemas/approachsettlement/1"         : "schemas/approachsettlement-v1.0.json",
+        "https://eddn.edcd.io/schemas/approachsettlement/1/test"    : "schemas/approachsettlement-v1.0.json",
+        "https://eddn.edcd.io/schemas/fssallbodiesfound/1"          : "schemas/fssallbodiesfound-v1.0.json",
+        "https://eddn.edcd.io/schemas/fssallbodiesfound/1/test"     : "schemas/fssallbodiesfound-v1.0.json",
+
+        "https://eddn.edcd.io/schemas/fssbodysignals/1"             : "schemas/fssbodysignals-v1.0.json",
+        "https://eddn.edcd.io/schemas/fssbodysignals/1/test"        : "schemas/fssbodysignals-v1.0.json",
+
+        "https://eddn.edcd.io/schemas/fsssignaldiscovered/1"            : "schemas/fsssignaldiscovered-v1.0.json",
+        "https://eddn.edcd.io/schemas/fsssignaldiscovered/1/test"       : "schemas/fsssignaldiscovered-v1.0.json",
+
+        "https://eddn.edcd.io/schemas/fcmaterials_journal/1"                : "schemas/fcmaterials_journal-v1.0.json",
+        "https://eddn.edcd.io/schemas/fcmaterials_journal/1/test"           : "schemas/fcmaterials_journal-v1.0.json",
+
+        "https://eddn.edcd.io/schemas/fcmaterials_capi/1"                : "schemas/fcmaterials_capi-v1.0.json",
+        "https://eddn.edcd.io/schemas/fcmaterials_capi/1/test"           : "schemas/fcmaterials_capi-v1.0.json",
+
+        "https://eddn.edcd.io/schemas/dockinggranted/1"               : "schemas/dockinggranted-v1.0.json",
+        "https://eddn.edcd.io/schemas/dockinggranted/1/test"          : "schemas/dockinggranted-v1.0.json",
+        "https://eddn.edcd.io/schemas/dockingdenied/1"                : "schemas/dockingdenied-v1.0.json",        
+        "https://eddn.edcd.io/schemas/dockingdenied/1/test"           : "schemas/dockingdenied-v1.0.json",        
     }
 
     GATEWAY_OUTDATED_SCHEMAS                = [
@@ -134,15 +155,14 @@ class _Settings(object):
 Settings = _Settings()
 
 
-def loadConfig():
-    '''
-    Loads in a settings file specified on the commandline if one has been specified.
+def loadConfig(cl_args):
+    """
+    Load in a commandline-specified settings file, if applicable.
+
     A convenience method if you don't need other things specified as commandline
     options. Otherwise, point the filename to Settings.loadFrom().
-    '''
-    parser = argparse.ArgumentParser()
-    parser.add_argument("-c", "--config", nargs="?", default=None)
-    args = parser.parse_args()
 
-    if args.config:
-        Settings.loadFrom(args.config)
+    :param cl_args: An `argparse.parse_args()` return.
+    """
+    if cl_args.config:
+        Settings.loadFrom(cl_args.config)