Merge branch 'master' of github.com:EDCD/EDDN

README.md

@@ -1,7 +1,7 @@
 # EDDN - Elite Dangerous Data Network
 
 ## About EDDN
-Elite Dangerous Data Network is a tool that facilitates the players of the game
+Elite Dangerous Data Network is a tool that facilitates players of the game
 [Elite Dangerous](https://www.elitedangerous.com/), including its 
 expansions, sharing data about the game galaxy with others.
 By pooling data in a common format, tools and analyses can be produced that add
@@ -15,17 +15,47 @@ archive or "current state" available to anyone.  What it provides is a
 stream of live data to any interested parties.  Some of those then make 
 aggregated data available for general use.
 
+---
 ---
 
 ## Using EDDN
 ### Game players
-If you are a player of the game and only want to help out by sharing the 
-data available to you over EDDN then please consult the
-[EDCD Cmdr's Guide](https://edcd.github.io/cmdrs-guide.html).  For the most 
-part if you want to share data then you will need to be playing the game on a 
-PC, but there are some tools that utilise an API provided by the game 
+It might be useful to consult the [EDCD Cmdr's Guide](https://edcd.github.io/cmdrs-guide.html)
+for a general overview of how players can contribute and use game data.
+
+---
+
+#### Contributing data
+There are a variety of tools available to players in order for them to help 
+out by contributing data, and to then utilise that data to enhance their 
+gameplay experience.
+
+For the most part any player who wishes to share data
+will need to be playing the game on a PC, as that gives direct access to 
+"Journal" files written by the game client.  These are the best source of 
+game data.
+There are, however, some tools that utilise an API provided by the game
 developer that can supply some data if you are playing on a console.
 
+So, on PC, look into installing one of the following tools:
+
+- [E:D Market Connector](https://github.com/EDCD/EDMarketConnector/wiki)
+- [EDDI](https://github.com/EDCD/EDDI)
+- [EDDiscovery](https://github.com/EDDiscovery/EDDiscovery)
+- [Elite Log Agent](https://github.com/DarkWanderer/Elite-Log-Agent)
+
+This list is not exhaustive, or intended to particular endorse any of these 
+projects over another, listed here or not.
+
+If you are playing on console some options are:
+
+- The 'console updater', available in a user's 'dashboard' on
+  [EDSM](https://www.edsm.net).
+- [Journal Limpet](https://journal-limpet.com/).
+
+---
+
+#### Utilising data
 If you're looking for tools that utilise EDDN data to enhance your experience
 then you're probably looking for one of the sites listed below.  NB: These are
 listed in name-alphabetical order and no particular ranking or endorsement is
@@ -43,9 +73,14 @@ intended.
   a 'Neutron Star' route plotter, but has since expanded into offering many 
   other route plotting tools and general data searching.
 
-There are many other third-party tools for Elite Dangerous listed on
-[Elite: Dangerous Codex](https://edcodex.info/), some of which will 
-interact with EDDN.  Check the [EDDN tag](https://edcodex.info/?m=tools&cat=9).
+---
+
+There are many other third-party tools for Elite Dangerous, both for 
+contributing data and utilising it, listed on
+[Elite: Dangerous Codex](https://edcodex.info/).  Some of them
+interact with EDDN - check the [EDDN tag](https://edcodex.info/?m=tools&cat=9).
+
+---
 
 ### Developers
 If you are a developer of a third-party tool that could be enhanced by 
@@ -55,18 +90,25 @@ uploading data to EDDN then please consult
 **DO NOT** assume that any code or documentation in the `master` (or 
 any other) branch on GitHub truly reflects the current live service!
 
-### Misc
-There is also a [wiki page](https://github.com/EDSM-NET/EDDN/wiki), but its 
-contents are currently being migrated into the source code tree (so that 
-they always match the in-use code).
+Anyone planning to send data too EDDN **MUST** comply with all the advice in
+that document, and the individual schema README files as applicable.  It's
+also the best resource for those listening to the EDDN data stream.
 
+---
+---
+
+## Misc
+
+### Service Status
 Consult [EDDN Status](https://eddn.edcd.io/) for some information about, 
 and statistics for, the live service.
 
----
-
-## Hosting of the live service
+### Hosting of the live service
 
 Hosting is currently provided by the
 [Elite: Dangerous Community Developers](https://edcd.github.io/).
 
+### Contacting the EDDN team
+
+* [EDCD Discord](https://discord.gg/XBsdCq9) - **Use the `#eddn` channel**.
+* [E:D forum thread](https://forums.frontier.co.uk/threads/elite-dangerous-data-network-eddn.585701/#post-9400060)

scripts/testing/gateway-responses/scan-valid_but-value-looks-like-formencoding.json

@@ -0,0 +1,37 @@
+{
+  "$schemaRef": "https://eddn.edcd.io/schemas/journal/1",
+  "message": {
+    "timestamp":"2021-11-05T15:46:28Z",
+    "event":"Scan",
+    "ScanType":"AutoScan",
+    "BodyName":"Elphin=wobble& Something",
+    "BodyID":1,
+    "Parents":[ {"Null":0} ],
+    "StarSystem":"Elphin=bloop; wibble",
+    "StarPos":[-30.12500,8.18750,-17.00000],
+    "SystemAddress":3932076118738,
+    "DistanceFromArrivalLS":0.000000,
+    "StarType":"K",
+    "Subclass":3,
+    "StellarMass":0.769531,
+    "Radius":587464832.000000,
+    "AbsoluteMagnitude":6.294067,
+    "Age_MY":9558,
+    "SurfaceTemperature":4796.000000,
+    "Luminosity":"V",
+    "SemiMajorAxis":1704360246658.325195,
+    "Eccentricity":0.348740,
+    "OrbitalInclination":-72.647343,
+    "Periapsis":86.347190,
+    "OrbitalPeriod":7189218699.932098,
+    "AscendingNode":0.000000,
+    "MeanAnomaly":351.262353,
+    "RotationPeriod":248957.736717,
+    "AxialTilt":-0.126915
+  },
+  "header": {
+    "uploaderID": "from Athanasius Testing",
+    "softwareName": "Athanasius Testing script",
+    "softwareVersion": "v0.0.1"
+  }
+}

scripts/testing/gateway-responses/test-bad-gzip.py

@@ -29,7 +29,7 @@ with open(sys.argv[1], 'r') as f:
   # This apparently causes compression to actually happen
   s.headers['Content-Encoding'] = 'gzip'
   r = s.post(
-    'https://beta.eddn.edcd.io:4431/upload/',
+    'https://dev.eddn.edcd.io:4432/upload/',
     data=msg,
   )
 

scripts/testing/gateway-responses/test-bad-gzip.sh

@@ -3,4 +3,4 @@
 # python `requests` appears to perform compression when you set the
 # 'Content-Encoding: gzip' header, so do this with curl.
 
-curl --verbose -d 'wegiuweuygtfawgep9aqe8fpq2387lfbr;iufvypq38764tpgf' -H 'Content-Encoding: gzip' 'https://beta.eddn.edcd.io:4431/upload/'
+curl --verbose -d 'wegiuweuygtfawgep9aqe8fpq2387lfbr;iufvypq38764tpgf' -H 'Content-Encoding: gzip' 'https://dev.eddn.edcd.io:4432/upload/'

scripts/testing/gateway-responses/test-gzip-bad-formdata.py

@@ -25,7 +25,7 @@ with open(sys.argv[1], 'r') as f:
   # Send that compressed data as a POST body
   r = http.request(
     'POST',
-    'https://beta.eddn.edcd.io:4431/upload/',
+    'https://dev.eddn.edcd.io:4432/upload/',
     headers={
       'Content-Encoding': 'gzip'
     },

scripts/testing/gateway-responses/test-gzip-correct-formdata.py

@@ -25,7 +25,7 @@ with open(sys.argv[1], 'r') as f:
   # Send that compressed data as a POST body
   r = http.request(
     'POST',
-    'https://beta.eddn.edcd.io:4431/upload/',
+    'https://dev.eddn.edcd.io:4432/upload/',
     headers={
       'Content-Encoding': 'gzip'
     },

scripts/testing/gateway-responses/test-gzip-plain-json.py

@@ -0,0 +1,34 @@
+#!/usr/bin/env python3
+
+import json
+import requests
+import sys
+import urllib3
+import zlib
+
+if len(sys.argv) != 2:
+  print('test-sender.py <filename>')
+  sys.exit(-1)
+
+with open(sys.argv[1], 'r') as f:
+  # Read from provided file
+  msg = f.read()
+
+  # Compress it
+  msg_gzip = zlib.compress(msg.encode('utf-8'))
+
+  http = urllib3.PoolManager()
+
+  # Send that compressed data as a POST body
+  r = http.request(
+    'POST',
+    'https://dev.eddn.edcd.io:4432/upload/',
+    headers={
+      'Content-Encoding': 'gzip'
+    },
+    body=msg_gzip
+  )
+
+  print(f'Response: {r.status!r}')
+  print(f'Body:\n{r.data.decode()}\n')
+

scripts/testing/gateway-responses/test-plain-bad-formdata.py

@@ -20,7 +20,7 @@ with open(sys.argv[1], 'r') as f:
   # Send that data as a POST body
   r = http.request(
     'POST',
-    'https://beta.eddn.edcd.io:4431/upload/',
+    'https://dev.eddn.edcd.io:4432/upload/',
     body=msg
   )
 

scripts/testing/gateway-responses/test-plain-correct-formdata.py

@@ -20,7 +20,7 @@ with open(sys.argv[1], 'r') as f:
   # Send that data as a POST body
   r = http.request(
     'POST',
-    'https://beta.eddn.edcd.io:4431/upload/',
+    'https://dev.eddn.edcd.io:4432/upload/',
     body=msg
   )
 

scripts/testing/gateway-responses/test-sender.py

@@ -13,7 +13,7 @@ with open(sys.argv[1], 'r') as f:
 
   s = requests.Session()
 
-  r = s.post('https://beta.eddn.edcd.io:4431/upload/', data=msg)
+  r = s.post('https://dev.eddn.edcd.io:4432/upload/', data=msg)
 
   print(f'Response: {r!r}')
   print(f'Body: {r.content.decode()}')

src/eddn/Gateway.py

@@ -166,23 +166,13 @@ def get_decompressed_message():
     else:
         logger.debug('Content-Encoding indicates *not* compressed...')
 
-        form_enc_parsed = urlparse.parse_qs(request.body.read())
-        if form_enc_parsed:
-            logger.debug('Request is form-encoded')
-
-            # Uncompressed request. Bottle handles all of the parsing of the
-            # POST key/vals, or un-encoded body.
-            data_key = request.forms.get('data')
-            if data_key:
-                logger.debug('form-encoded POST request detected...')
-                # This is a form-encoded POST. Support the silly people.
-                message_body = data_key
-    
-            else:
-                raise MalformedUploadError(
-                    "No 'data' POST key/value found. Check your POST key "
-                    "name for spelling, and make sure you're passing a value."
-                )
+        # Uncompressed request. Bottle handles all of the parsing of the
+        # POST key/vals, or un-encoded body.
+        data_key = request.forms.get('data')
+        if data_key:
+            logger.debug('form-encoded POST request detected...')
+            # This is a form-encoded POST. Support the silly people.
+            message_body = data_key
 
         else:
             logger.debug('Plain POST request detected...')