Merge branch 'master' of github.com:EDCD/EDDN

schemas/README-EDDN-schemas.md

@@ -62,13 +62,27 @@ the available endpoints and how they work. There is some
 [third-party documentation](https://github.com/Athanasius/fd-api/blob/main/docs/README.md)
 by Athanasius.
 
+It is *not* recommended to use CAPI data as the source as it's fraught with
+additional issues.  EDMarketConnector does so in order to facilitate
+obtaining data without the player needing to open the commodities screen.
+
+#### Detecting CAPI data lag
+
 When using the Companion API please be aware that the server that supplies this
 data sometimes lags behind the game - usually by a few seconds, sometimes by
 minutes. You MUST check in the data from the CAPI that the Cmdr is
-docked (`["commander"]["docked"]` is `True`) and that the station and
+docked, and that the station and system names match those
-system (`["lastStarport"]["name"]` and `["lastSystem"]["name"]`) match those
 reported from the Journal before using the data for the commodity, outfitting
-and shipyard schemas.
+and shipyard schemas:
+
+1. Retrieve the commander data from the `/profile` CAPI endpoint.
+2. Check that `commander['docked']` is true.  If not, abort.
+3. Retrieve the data from the `/market` and `/shipyard` CAPI endpoints.
+4. Compare the system and station name from the CAPI market data,
+   `["lastStarport"]["name"]` and `["lastSystem"]["name"]`,
+   to that from the last `Docked` or `Location` journal event.  If either does
+   not match then you MUST **abort**.  This likely indicates that the CAPI 
+   data is lagging behind the game client state and thus should not be used.
 
 ---
 
@@ -170,39 +184,46 @@ Each `message` object must have, at bare minimum:
    much more than this. Again, consult the
    [schemas and their documentation](./).
 
+Note that many of the key names chosen in the schemas are based on the CAPI 
+data, not Journal events, because the CAPI came first.  This means renaming 
+many of the keys from Journal events to match the schema.
+
 EDDN is intended to transport generic data not specific to any particular Cmdr
 and to reflect the data that a player would see in-game in station services or
 the local map. To that end, uploading applications MUST ensure that messages do
 not contain any Cmdr-specific data (other than "uploaderID" and the "horizons"
-flag). In practice as of E:D 3.3 this means:
+flag).
 
-* commodity: Skip commodities with `"categoryname": "NonMarketable"` (i.e.
+The individual schemas will instruct you on various elisions (removals) to 
-  Limpets - not purchasable in station market) or `"legality":` *non-empty
+be made to comply with this.
-  string* (not normally traded at this station market).
-* outfitting: Skip items whose availability depends on the Cmdr's status rather
-  than on the station. Namely:
-    - Items that aren't weapons/utilities (`Hpt_*`), standard/internal
-      modules (`Int_*`) or armour (`*_Armour_*`) (i.e. bobbleheads, decals,
-      paintjobs and shipkits).
-    - Items that have a non-null `"sku"` property, unless
-      it's `"ELITE_HORIZONS_V_PLANETARY_LANDINGS"` (i.e. PowerPlay and tech
-      broker items).
-    - The `"Int_PlanetApproachSuite"` module (for historical reasons).
-* shipyard: *Include* ships listed in the `"unavailable_list"` property (i.e.
-  available at this station, but not to this Cmdr).
-* journal: Strip out `"..._Localised"` properties throughout the data
-  structure.
-* journal/Docked: Strip out `"Wanted"`, `"ActiveFine"`, `"CockpitBreach"`
-  properties
-* journal/FSDJump: Strip out `"Wanted"`, `"BoostUsed"`, `"FuelLevel"`
-  , `"FuelUsed"` and `"JumpDist"` properties.
-* journal/Location: Strip out `"Wanted"`, `"Latitude"` and `"Longitude"`
-  properties.
-* journal/Location and journal/FSDJump: strip out `"HappiestSystem"`
-  , `"HomeSystem"`, `"MyReputation"` and `"SquadronFaction"` properties within
-  the list of `"Factions"`.
 
 Some of these requirements are also enforced by the schemas, and some things
 the schemas enforce might not be explicitly called out here, so **do**
 check what you're sending against the schema when implementing sending new
 events.
+
+## Receiving messages
+
+EDDN provides a continuous stream of information from uploaders. To use this
+data you'll need to connect to the stream using ZeroMQ (a library is probably
+available for your language of choice).
+
+The URL for the live Relay is:
+
+    tcp://eddn.edcd.io:9500
+
+Once you've connected to that you will receive messages.  To access the 
+data you will first need to zlib-decompress each message.  Then you will 
+have a textual JSON object as per the schemas.
+
+In general, check the guidance for [Uploading messages](#uploading-messages)
+for the expected format of the messages.
+
+Consumers can utilise the `$schemaRef` value to determine which schema a 
+particular message is for.  There is no need to validate the messages 
+against the schemas yourself, as that is performed on the EDDN Gateway.  
+Messages that do not pass the schema validation there are not forwarded to 
+receivers.
+
+There is [example code](https://github.com/EDCD/EDDN/tree/master/examples)
+available for a variety of programming languages to help you get started.

schemas/blackmarket-README.md

@@ -0,0 +1,55 @@
+# EDDN BlackMarket Schema
+
+## Introduction
+
+**This schema is deprecated.  The same data has been made available via the
+`prohibited` array in the [commodity schema](./commodity-v3.0.json) since 
+September 2017.**
+
+What follows below is only for historical curiosity.
+
+---
+
+Here we document how to take data from an ED `MarketSell` Journal Event and
+properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+## Senders
+The primary data source for this schema is the ED Journal event `MarketSell`.
+
+### Key Renames
+#### name
+Due to how the EDDN schema is defined the `Type` key/value should
+have the key renamed to `name`.
+
+#### prohibited
+Due to how the EDDN schema is defined the `IllegalGoods` key/value should
+have the key renamed to `prohibited`.
+
+#### marketID
+The Journal documentation says this is `MarketID`, but in the schema the 
+`m` is lower case.
+
+### Elisions
+You MUST remove the following key/value pairs from the data:
+
+  - `TotalSale`
+  - `AvgPricePaid`
+  - `StolenGoods`
+  - `BlackMarket` - Because we're using this schema, so this is un-necessary. 
+
+### Augmentations
+#### horizons flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### odyssey flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### systemName
+The star system name for where this market is.   Use the `StarSystem` value
+from the prior `Docked` or `Location` event.
+
+#### stationName
+From the `StationName` value on the prior `Docked` or `Location` event.

schemas/codexentry-README.md

@@ -17,6 +17,12 @@ You MUST remove any key where the key name ends in
 You MUST remove the two keys `IsNewEntry` and `NewTraitsDiscovered`.
 
 ### Augmentations
+#### horizons flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### odyssey flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
 #### StarPos
 You MUST **add** a `StarPos` key with value of type `array` containing the 
 galaxy co-ordinates of the system.  You will need to have obtained these 
@@ -58,7 +64,7 @@ release, Update 7, plus one patch).
    away.
 5. If Status.json does **not** have `BodyName` then clear `status_body_name`.
 6. For a `CodexEntry` event:
-    1. Check that `status_body_name` is set. If it is not, exit.
+    1. Only if `status_body_name` is set:
         1. Set the EDDN `codexentry` schema message `BodyName` to this value.
         2. Check if it matches the `journal_body_name` value, and 
            ONLY if they match, set `BodyID` in the EDDN `codexentry`

schemas/commodity-README.md

@@ -0,0 +1,94 @@
+# EDDN Commodity Schema
+
+## Introduction
+Here we document how to take data from an ED `Market` Journal Event and
+properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+## Senders
+The primary data source for this schema is the ED Journal event `Market`, 
+and the additional file, `Market.json`, that it signals the writing of.
+
+So, look for the `Market` event, and when it occurs open and read the 
+`Market.json` file which contains the actual data.  Treat *that* data as 
+the event.
+
+It *is* also possible to construct this data from a CAPI `/market` query.  
+See [Using CAPI data](#using-capi-data) below.
+
+### Statusflags
+`statusFlags` is an optional augmentation that can be used to identify
+`Producer`, `Consumer`, and `Rare` goods from the output of `Market.json`.  
+So, as per the schema, do include it if available.
+
+### Key Renames
+Many of the key names have a different case defined in this schema, make 
+sure you are renaming them as appropriate.
+
+#### StarSystem to systemName
+Rename the `StarSystem` key name to `systemName`.
+
+### Elisions
+#### Remove _Localised key/values
+All keys whose name ends with `_Localised`, i.e. the `Name_Localised`
+key/values in Items.
+
+#### Other Elisions
+You MUST remove the following key/value pairs from the data:
+
+- `StationType` key/value.
+- `Producer` key/value pair in Items.
+- `Rare` key/value pair in Items.
+- `id` key/value pair in Items.
+
+In the list of commodites:
+
+- Skip commodities with `"categoryname": "NonMarketable"` (i.e.
+  Limpets - not purchasable in station market) or a *non-empty*`"legality":` 
+  string (not normally traded at this station market).
+
+#### Item Category
+Remove not only the `Category_Localised` key/value, as above, but also the
+`Category` key/value pair from each Item.
+
+### Augmentations
+#### horizons flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### odyssey flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+### Using CAPI data
+It is *not* recommended to use CAPI data as the source as it's fraught with 
+additional issues.  EDMarketConnector does so in order to facilitate 
+obtaining data without the player needing to open the commodities screen.
+
+Please read
+[the guidance on checking for CAPI lag](README-EDDN-schemas.md#detecting-capi-data-lag)
+before utilising CAPI data for EDDN messages.
+
+Note that CAPI `/market` data will sometimes have the `StatusFlasg` per 
+item, which are defined as optional in this schema (because they're not in 
+the Market.json data).  You SHOULD include this data in your message if 
+using CAPI as the source.
+
+Now you will need to construct the necessary additional fields:
+
+#### CAPI horizons flag
+If your application can be certain that the game client is still running, 
+and logged into the game (not just run to the main menu), then you can 
+simply use the value from the `LoadGame` journal event.
+
+Otherwise, you MUST check if any of the economies from the `/market` 
+data have a `name` of `Colony`, if so, set this flag true.
+
+Additionally, you should retrieve the CAPI `/shipyard` endpoint and check if 
+any of the listed ships or modules have a `sku` value of 
+`ELITE_HORIZONS_V_PLANETARY_LANDINGS`.  If so, set this flag true.
+
+#### CAPI odyssey flag
+Unfortunately there is no method to be *certain* of this from CAPI data, so 
+you will have to trust in the system/station name check and use the value 
+from the Journal `LoadGame` event.

schemas/fssdiscoveryscan-README.md

@@ -0,0 +1,32 @@
+# EDDN FSSDiscoveryScan Schema
+
+## Introduction
+Here we document how to take data from an ED `FSSDiscoveryScan` Journal 
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+## Senders
+The primary data source for this schema is the ED Journal event 
+`FSSDiscoveryScan`.
+
+### Key Renames
+Many of the key names have a different case defined in this schema, make 
+sure you are renaming them as appropriate.
+
+### Elisions
+You MUST remove the following key/value pairs from the data:
+
+  - `Progress` key/value pair.
+
+### Augmentations
+#### horizons flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### odyssey flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### StarPos
+You MUST add a `StarPos` array containing the system co-ordinates from the 
+last `FSDJump`, `CarrierJump`, or `Location` event.

schemas/journal-README.md

@@ -0,0 +1,86 @@
+# EDDN Journal Schema
+
+## Introduction
+Here we document how to take data from miscellaneous ED Journal
+events and properly structure it for sending to EDDN.
+
+This is the historical "all Journal events" schema that will be deprecated 
+in the future.  Please check for a schema specific to the journal event 
+under consideration to see if data should be sent on that event specific 
+schema instead.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+## Senders
+The primary data source for this schema is the ED Journal events:
+
+  - `Docked`
+  - `FSDJump`
+  - `Scan`
+  - `Location`
+  - `SAASignalsFound`
+  - `CarrierJump`
+  - `CodexEntry` - But see the separate
+    [codexentry schema](./codexentry-README.md) documentation.
+
+### Key Renames
+Many of the key names have a different case defined in this schema, make
+sure you are renaming them as appropriate.
+
+### Elisions
+#### Remove _Localised key/values
+All keys whose name ends with `_Localised`, i.e. the `Name_Localised`
+key/values in Items.
+
+#### Personal data in `Docked` events
+The following keys+values should be removed from `Docked` event data:
+
+  - `Wanted`
+  - `ActiveFine`
+  - `CockpitBreach`
+
+#### Personal data in `FSDJump` events
+The following keys+values should be removed from `FSDJump` event data:
+
+- `Wanted`
+- `BoostUsed`
+- `FuelLevel`
+- `FuelUsed`
+- `JumpDist`
+- `HappiestSystem` from within the list of `Factions`.
+- `HomeSystem` from within the list of `Factions`.
+- `MyReputation` from within the list of `Factions`.
+- `SquadronFaction` from within the list of `Factions`.
+
+####  Personal data in `Location` events
+The following keys+values should be removed from `Location` event data:
+
+- `Wanted`
+- `Latitude`
+- `Longitude`
+- `HappiestSystem` from within the list of `Factions`.
+- `HomeSystem` from within the list of `Factions`.
+- `MyReputation` from within the list of `Factions`.
+- `SquadronFaction` from within the list of `Factions`.
+
+### Augmentations
+#### horizons flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### odyssey flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### StarSystem
+You MUST add a `StarSystem` key/value pair representing the name of the 
+system this event occurred in.  Source this from either `Location`, 
+`FSDJump` or `CarrierJump` as appropriate.
+
+#### SystemAddress
+You MUST add a `SystemAddress` key/value pair representing the numerical ID 
+of the system this event occurred in.  Source this from either `Location`,
+`FSDJump` or `CarrierJump` as appropriate.
+
+#### StarPos
+You MUST add a `StarPos` array containing the system co-ordinates from the
+last `FSDJump`, `CarrierJump`, or `Location` event.

schemas/navbeaconscan-README.md

@@ -0,0 +1,31 @@
+# EDDN NavBeaconScan Schema
+
+## Introduction
+Here we document how to take data from an ED `NavBeaconScan` Journal
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+## Senders
+The primary data source for this schema is the ED Journal event
+`NavBeaconScan`.
+
+### Elisions
+There are no elisions in this schema.
+
+### Augmentations
+#### horizons flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### odyssey flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### StarSystem
+You MUST add a `StarSystem` key/value pair representing the name of the
+system this event occurred in.  Source this from either `Location`,
+`FSDJump` or `CarrierJump` as appropriate.
+
+#### StarPos
+You MUST add a `StarPos` array containing the system co-ordinates from the
+last `FSDJump`, `CarrierJump`, or `Location` event.

schemas/navroute-README.md

@@ -0,0 +1,29 @@
+# EDDN NavRoute Schema
+
+## Introduction
+Here we document how to take data from an ED `NavRoute` Journal
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+## Senders
+The primary data source for this schema is the `NavRoute.json` file.  That 
+it has been freshly written is signalled by the ED Journal event `NavRoute`.
+
+So, monitor the Journal as normal, and when you see a `NavRoute` event open 
+the `NavRoute.json` file for reading, read it, and close it again.  Use the 
+data you got from reading this file, not merely the Journal event.
+
+The primary data to be sent is the `Route` array from the contents of the 
+separate file.
+
+### Elisions
+There are no elisions in this schema.
+
+### Augmentations
+#### horizons flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### odyssey flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.

schemas/outfitting-README.md

@@ -0,0 +1,42 @@
+# EDDN Outfitting Schema
+
+## Introduction
+Here we document how to take data from an ED `Outfitting` Journal
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+## Senders
+The primary data source for this schema is the ED Journal event
+`Outfitting`.
+
+You MAY also source this data from the CAPI `/shipyard` endpoint.
+Please read
+[the guidance on checking for CAPI lag](README-EDDN-schemas.md#detecting-capi-data-lag)
+before utilising CAPI data for EDDN messages.
+
+You only need the `name` key's value for each member of the `modules` array.
+
+### Key Renames
+Many of the key names have a different case defined in this schema, make
+sure you are renaming them as appropriate.
+
+### Elisions
+Remove items whose availability depends on the Cmdr's status rather than on the
+station. Namely:
+
+- Items that aren't weapons/utilities (`Hpt_*`), standard/internal
+  modules (`Int_*`) or armour (`*_Armour_*`) (i.e. bobbleheads, decals,
+  paintjobs and shipkits).  This is enforced by the schema.
+- Items that have a non-null `"sku"` property, unless
+  it's `"ELITE_HORIZONS_V_PLANETARY_LANDINGS"` (i.e. PowerPlay and tech
+  broker items).
+- The `"Int_PlanetApproachSuite"` module (for historical reasons).
+
+### Augmentations
+#### horizons flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### odyssey flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.

schemas/scanbarycentre-README.md

@@ -0,0 +1,26 @@
+# EDDN ScanBaryCentre Schema
+
+## Introduction
+Here we document how to take data from an ED `ScanBaryCentre` Journal
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+## Senders
+The primary data source for this schema is the ED Journal event
+`ScanBaryCentre`.
+
+### Elisions
+There are no elisions in this schema.
+
+### Augmentations
+#### horizons flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### odyssey flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### StarPos
+You MUST add a `StarPos` array containing the system co-ordinates from the
+last `FSDJump`, `CarrierJump`, or `Location` event.

schemas/shipyard-README.md

@@ -0,0 +1,40 @@
+# EDDN Shipyard Schema
+
+## Introduction
+Here we document how to take data from an ED `Shipyard` Journal
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+## Senders
+The primary data source for this schema is the ED Journal event
+`Shipyard`.
+
+You MAY also source this data from the CAPI `/shipyard` endpoint.
+Please read
+[the guidance on checking for CAPI lag](README-EDDN-schemas.md#detecting-capi-data-lag)
+before utilising CAPI data for EDDN messages.
+
+ You only need the `name` key's value for each member of the `PriceList` 
+array (if using Journal, it will be from the `ships` array if using CAPI 
+data).
+
+When using CAPI data *include* ships listed in the `"unavailable_list"` 
+property (i.e. available at this station, but not to this Cmdr).
+
+This list of ship names will go in the `ships` array in the EDDN message.
+
+### Key Renames
+Many of the key names have a different case defined in this schema, make
+sure you are renaming them as appropriate.
+
+### Elisions
+There are no elisions in this schema.
+
+### Augmentations
+#### horizons flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### odyssey flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.

src/eddn/Gateway.py

@@ -21,10 +21,23 @@ from eddn.core.Validator import Validator, ValidationSeverity
 
 from gevent import monkey
 monkey.patch_all()
+import bottle
 from bottle import Bottle, run, request, response, get, post
+bottle.BaseRequest.MEMFILE_MAX = 1024 * 1024 # 1MiB, default is/was 100KiB
 app = Bottle()
 
 logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+__logger_channel = logging.StreamHandler()
+__logger_formatter = logging.Formatter(
+    '%(asctime)s - %(levelname)s - %(module)s:%(lineno)d: %(message)s'
+)
+__logger_formatter.default_time_format = '%Y-%m-%d %H:%M:%S'
+__logger_formatter.default_msec_format = '%s.%03d'
+__logger_channel.setFormatter(__logger_formatter)
+logger.addHandler(__logger_channel)
+logger.info('Made logger')
+
 
 # This socket is used to push market data out to the Announcers over ZeroMQ.
 context = zmq.Context()
@@ -38,6 +51,37 @@ statsCollector = StatsCollector()
 statsCollector.start()
 
 
+def extract_message_details(parsed_message):
+    uploader_id = '<<UNKNOWN>>'
+    software_name = '<<UNKNOWN>>'
+    software_version = '<<UNKNOWN>>'
+    schema_ref = '<<UNKNOWN>>'
+    journal_event = '<<UNKNOWN>>'
+
+    if 'header' in parsed_message:
+        if 'uploaderID' in parsed_message['header']:
+            uploader_id = parsed_message['header']['uploaderID']
+
+        if 'softwareName' in parsed_message['header']:
+            software_name = parsed_message['header']['softwareName']
+
+        if 'softwareVersion' in parsed_message['header']:
+            software_version = parsed_message['header']['softwareVersion']
+
+    if '$schemaRef' in parsed_message:
+        schema_ref = parsed_message['$schemaRef']
+
+
+        if '/journal/' in schema_ref:
+            if 'message' in parsed_message:
+                if 'event' in parsed_message['message']:
+                    journal_event = parsed_message['message']['event']
+
+        else:
+            journal_event = '-'
+
+    return uploader_id, software_name, software_version, schema_ref, journal_event
+
 def configure():
     # Get the list of transports to bind from settings. This allows us to PUB
     # messages to multiple announcers over a variety of socket types
@@ -131,9 +175,24 @@ def parse_and_error_handle(data):
     ) as exc:
         # Something bad happened. We know this will return at least a
         # semi-useful error message, so do so.
+        try:
+            logger.error('Error - JSON parse failed (%d, "%s", "%s", "%s", "%s", "%s") from %s:\n%s\n' % (
+                    request.content_length,
+                    '<<UNKNOWN>>',
+                    '<<UNKNOWN>>',
+                    '<<UNKNOWN>>',
+                    '<<UNKNOWN>>',
+                    '<<UNKNOWN>>',
+                    get_remote_address(),
+                    data[:512]
+            ))
+
+        except Exception as e:
+            print('Logging of "JSON parse failed" failed: %s' % (e.message))
+            pass
+
         response.status = 400
-        logger.error("Error to %s: %s" % (get_remote_address(), exc.message))
+        return 'FAIL: ' + str(exc)
-        return str(exc)
 
     # Here we check if an outdated schema has been passed
     if parsed_message["$schemaRef"] in Settings.GATEWAY_OUTDATED_SCHEMAS:
@@ -149,11 +208,35 @@ def parse_and_error_handle(data):
 
         # Sends the parsed message to the Relay/Monitor as compressed JSON.
         gevent.spawn(push_message, parsed_message, parsed_message['$schemaRef'])
-        logger.info("Accepted %s upload from %s" % (
+
-            parsed_message, get_remote_address()
+        try:
-        ))
+            uploader_id, software_name, software_version, schema_ref, journal_event = extract_message_details(parsed_message)
+            logger.info('Accepted (%d, "%s", "%s", "%s", "%s", "%s") from %s' % (
+                request.content_length,
+                uploader_id, software_name, software_version, schema_ref, journal_event,
+                get_remote_address()
+            ))
+
+        except Exception as e:
+            print('Logging of Accepted request failed: %s' % (e.message))
+            pass
+
         return 'OK'
+
     else:
+        try:
+            uploader_id, software_name, software_version, schema_ref, journal_event = extract_message_details(parsed_message)
+            logger.error('Failed Validation "%s" (%d, "%s", "%s", "%s", "%s", "%s") from %s' % (
+                    str(validationResults.messages),
+                    request.content_length,
+                    uploader_id, software_name, software_version, schema_ref, journal_event,
+                    get_remote_address()
+            ))
+
+        except Exception as e:
+            print('Logging of Failed Validation failed: %s' % (e.message))
+            pass
+
         response.status = 400
         statsCollector.tally("invalid")
         return "FAIL: " + str(validationResults.messages)
@@ -164,17 +247,34 @@ def upload():
     try:
         # Body may or may not be compressed.
         message_body = get_decompressed_message()
+
     except zlib.error as exc:
         # Some languages and libs do a crap job zlib compressing stuff. Provide
         # at least some kind of feedback for them to try to get pointed in
         # the correct direction.
         response.status = 400
-        logger.error("gzip error with %s: %s" % (get_remote_address(), exc.message))
+        try:
+            logger.error('gzip error (%d, "%s", "%s", "%s", "%s", "%s") from %s' % (
+                    request.content_length,
+                    '<<UNKNOWN>>',
+                    '<<UNKNOWN>>',
+                    '<<UNKNOWN>>',
+                    '<<UNKNOWN>>',
+                    '<<UNKNOWN>>',
+                    get_remote_address()
+            ))
+
+        except Exception as e:
+            print('Logging of "gzip error" failed: %s' % (e.message))
+            pass
+
         return exc.message
+
     except MalformedUploadError as exc:
         # They probably sent an encoded POST, but got the key/val wrong.
         response.status = 400
-        logger.error("Error to %s: %s" % (get_remote_address(), exc.message))
+        logger.error("MalformedUploadError from %s: %s" % (get_remote_address(), exc.message))
+
         return exc.message
 
     statsCollector.tally("inbound")