diff --git a/README.md b/README.md index 42978ee..4127a10 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # EDDN - Elite Dangerous Data Network ## About EDDN -Elite Dangerous Data Network is a tool that facilitates the players of the game +Elite Dangerous Data Network is a tool that facilitates players of the game [Elite Dangerous](https://www.elitedangerous.com/), including its expansions, sharing data about the game galaxy with others. By pooling data in a common format, tools and analyses can be produced that add @@ -15,17 +15,47 @@ archive or "current state" available to anyone. What it provides is a stream of live data to any interested parties. Some of those then make aggregated data available for general use. +--- --- ## Using EDDN ### Game players -If you are a player of the game and only want to help out by sharing the -data available to you over EDDN then please consult the -[EDCD Cmdr's Guide](https://edcd.github.io/cmdrs-guide.html). For the most -part if you want to share data then you will need to be playing the game on a -PC, but there are some tools that utilise an API provided by the game +It might be useful to consult the [EDCD Cmdr's Guide](https://edcd.github.io/cmdrs-guide.html) +for a general overview of how players can contribute and use game data. + +--- + +#### Contributing data +There are a variety of tools available to players in order for them to help +out by contributing data, and to then utilise that data to enhance their +gameplay experience. + +For the most part any player who wishes to share data +will need to be playing the game on a PC, as that gives direct access to +"Journal" files written by the game client. These are the best source of +game data. +There are, however, some tools that utilise an API provided by the game developer that can supply some data if you are playing on a console. +So, on PC, look into installing one of the following tools: + +- [E:D Market Connector](https://github.com/EDCD/EDMarketConnector/wiki) +- [EDDI](https://github.com/EDCD/EDDI) +- [EDDiscovery](https://github.com/EDDiscovery/EDDiscovery) +- [Elite Log Agent](https://github.com/DarkWanderer/Elite-Log-Agent) + +This list is not exhaustive, or intended to particular endorse any of these +projects over another, listed here or not. + +If you are playing on console some options are: + +- The 'console updater', available in a user's 'dashboard' on + [EDSM](https://www.edsm.net). +- [Journal Limpet](https://journal-limpet.com/). + +--- + +#### Utilising data If you're looking for tools that utilise EDDN data to enhance your experience then you're probably looking for one of the sites listed below. NB: These are listed in name-alphabetical order and no particular ranking or endorsement is @@ -43,9 +73,14 @@ intended. a 'Neutron Star' route plotter, but has since expanded into offering many other route plotting tools and general data searching. -There are many other third-party tools for Elite Dangerous listed on -[Elite: Dangerous Codex](https://edcodex.info/), some of which will -interact with EDDN. Check the [EDDN tag](https://edcodex.info/?m=tools&cat=9). +--- + +There are many other third-party tools for Elite Dangerous, both for +contributing data and utilising it, listed on +[Elite: Dangerous Codex](https://edcodex.info/). Some of them +interact with EDDN - check the [EDDN tag](https://edcodex.info/?m=tools&cat=9). + +--- ### Developers If you are a developer of a third-party tool that could be enhanced by @@ -55,18 +90,25 @@ uploading data to EDDN then please consult **DO NOT** assume that any code or documentation in the `master` (or any other) branch on GitHub truly reflects the current live service! -### Misc -There is also a [wiki page](https://github.com/EDSM-NET/EDDN/wiki), but its -contents are currently being migrated into the source code tree (so that -they always match the in-use code). +Anyone planning to send data too EDDN **MUST** comply with all the advice in +that document, and the individual schema README files as applicable. It's +also the best resource for those listening to the EDDN data stream. +--- +--- + +## Misc + +### Service Status Consult [EDDN Status](https://eddn.edcd.io/) for some information about, and statistics for, the live service. ---- - -## Hosting of the live service +### Hosting of the live service Hosting is currently provided by the [Elite: Dangerous Community Developers](https://edcd.github.io/). +### Contacting the EDDN team + +* [EDCD Discord](https://discord.gg/XBsdCq9) - **Use the `#eddn` channel**. +* [E:D forum thread](https://forums.frontier.co.uk/threads/elite-dangerous-data-network-eddn.585701/#post-9400060) diff --git a/scripts/testing/gateway-responses/scan-valid_but-value-looks-like-formencoding.json b/scripts/testing/gateway-responses/scan-valid_but-value-looks-like-formencoding.json new file mode 100644 index 0000000..49f7283 --- /dev/null +++ b/scripts/testing/gateway-responses/scan-valid_but-value-looks-like-formencoding.json @@ -0,0 +1,37 @@ +{ + "$schemaRef": "https://eddn.edcd.io/schemas/journal/1", + "message": { + "timestamp":"2021-11-05T15:46:28Z", + "event":"Scan", + "ScanType":"AutoScan", + "BodyName":"Elphin=wobble& Something", + "BodyID":1, + "Parents":[ {"Null":0} ], + "StarSystem":"Elphin=bloop; wibble", + "StarPos":[-30.12500,8.18750,-17.00000], + "SystemAddress":3932076118738, + "DistanceFromArrivalLS":0.000000, + "StarType":"K", + "Subclass":3, + "StellarMass":0.769531, + "Radius":587464832.000000, + "AbsoluteMagnitude":6.294067, + "Age_MY":9558, + "SurfaceTemperature":4796.000000, + "Luminosity":"V", + "SemiMajorAxis":1704360246658.325195, + "Eccentricity":0.348740, + "OrbitalInclination":-72.647343, + "Periapsis":86.347190, + "OrbitalPeriod":7189218699.932098, + "AscendingNode":0.000000, + "MeanAnomaly":351.262353, + "RotationPeriod":248957.736717, + "AxialTilt":-0.126915 + }, + "header": { + "uploaderID": "from Athanasius Testing", + "softwareName": "Athanasius Testing script", + "softwareVersion": "v0.0.1" + } +} diff --git a/scripts/testing/gateway-responses/test-bad-gzip.py b/scripts/testing/gateway-responses/test-bad-gzip.py index ff5ff11..9d50f1f 100644 --- a/scripts/testing/gateway-responses/test-bad-gzip.py +++ b/scripts/testing/gateway-responses/test-bad-gzip.py @@ -29,7 +29,7 @@ with open(sys.argv[1], 'r') as f: # This apparently causes compression to actually happen s.headers['Content-Encoding'] = 'gzip' r = s.post( - 'https://beta.eddn.edcd.io:4431/upload/', + 'https://dev.eddn.edcd.io:4432/upload/', data=msg, ) diff --git a/scripts/testing/gateway-responses/test-bad-gzip.sh b/scripts/testing/gateway-responses/test-bad-gzip.sh index befde64..77592bf 100644 --- a/scripts/testing/gateway-responses/test-bad-gzip.sh +++ b/scripts/testing/gateway-responses/test-bad-gzip.sh @@ -3,4 +3,4 @@ # python `requests` appears to perform compression when you set the # 'Content-Encoding: gzip' header, so do this with curl. -curl --verbose -d 'wegiuweuygtfawgep9aqe8fpq2387lfbr;iufvypq38764tpgf' -H 'Content-Encoding: gzip' 'https://beta.eddn.edcd.io:4431/upload/' +curl --verbose -d 'wegiuweuygtfawgep9aqe8fpq2387lfbr;iufvypq38764tpgf' -H 'Content-Encoding: gzip' 'https://dev.eddn.edcd.io:4432/upload/' diff --git a/scripts/testing/gateway-responses/test-gzip-bad-formdata.py b/scripts/testing/gateway-responses/test-gzip-bad-formdata.py index 0ae79fd..0e3e1f0 100644 --- a/scripts/testing/gateway-responses/test-gzip-bad-formdata.py +++ b/scripts/testing/gateway-responses/test-gzip-bad-formdata.py @@ -25,7 +25,7 @@ with open(sys.argv[1], 'r') as f: # Send that compressed data as a POST body r = http.request( 'POST', - 'https://beta.eddn.edcd.io:4431/upload/', + 'https://dev.eddn.edcd.io:4432/upload/', headers={ 'Content-Encoding': 'gzip' }, diff --git a/scripts/testing/gateway-responses/test-gzip-correct-formdata.py b/scripts/testing/gateway-responses/test-gzip-correct-formdata.py index 066f617..e58c1d9 100644 --- a/scripts/testing/gateway-responses/test-gzip-correct-formdata.py +++ b/scripts/testing/gateway-responses/test-gzip-correct-formdata.py @@ -25,7 +25,7 @@ with open(sys.argv[1], 'r') as f: # Send that compressed data as a POST body r = http.request( 'POST', - 'https://beta.eddn.edcd.io:4431/upload/', + 'https://dev.eddn.edcd.io:4432/upload/', headers={ 'Content-Encoding': 'gzip' }, diff --git a/scripts/testing/gateway-responses/test-gzip-plain-json.py b/scripts/testing/gateway-responses/test-gzip-plain-json.py new file mode 100644 index 0000000..db7042a --- /dev/null +++ b/scripts/testing/gateway-responses/test-gzip-plain-json.py @@ -0,0 +1,34 @@ +#!/usr/bin/env python3 + +import json +import requests +import sys +import urllib3 +import zlib + +if len(sys.argv) != 2: + print('test-sender.py ') + sys.exit(-1) + +with open(sys.argv[1], 'r') as f: + # Read from provided file + msg = f.read() + + # Compress it + msg_gzip = zlib.compress(msg.encode('utf-8')) + + http = urllib3.PoolManager() + + # Send that compressed data as a POST body + r = http.request( + 'POST', + 'https://dev.eddn.edcd.io:4432/upload/', + headers={ + 'Content-Encoding': 'gzip' + }, + body=msg_gzip + ) + + print(f'Response: {r.status!r}') + print(f'Body:\n{r.data.decode()}\n') + diff --git a/scripts/testing/gateway-responses/test-plain-bad-formdata.py b/scripts/testing/gateway-responses/test-plain-bad-formdata.py index f0fccb4..04cc0bd 100644 --- a/scripts/testing/gateway-responses/test-plain-bad-formdata.py +++ b/scripts/testing/gateway-responses/test-plain-bad-formdata.py @@ -20,7 +20,7 @@ with open(sys.argv[1], 'r') as f: # Send that data as a POST body r = http.request( 'POST', - 'https://beta.eddn.edcd.io:4431/upload/', + 'https://dev.eddn.edcd.io:4432/upload/', body=msg ) diff --git a/scripts/testing/gateway-responses/test-plain-correct-formdata.py b/scripts/testing/gateway-responses/test-plain-correct-formdata.py index 99ccf63..a00815f 100644 --- a/scripts/testing/gateway-responses/test-plain-correct-formdata.py +++ b/scripts/testing/gateway-responses/test-plain-correct-formdata.py @@ -20,7 +20,7 @@ with open(sys.argv[1], 'r') as f: # Send that data as a POST body r = http.request( 'POST', - 'https://beta.eddn.edcd.io:4431/upload/', + 'https://dev.eddn.edcd.io:4432/upload/', body=msg ) diff --git a/scripts/testing/gateway-responses/test-sender.py b/scripts/testing/gateway-responses/test-sender.py index 8bd9cde..de563a7 100644 --- a/scripts/testing/gateway-responses/test-sender.py +++ b/scripts/testing/gateway-responses/test-sender.py @@ -13,7 +13,7 @@ with open(sys.argv[1], 'r') as f: s = requests.Session() - r = s.post('https://beta.eddn.edcd.io:4431/upload/', data=msg) + r = s.post('https://dev.eddn.edcd.io:4432/upload/', data=msg) print(f'Response: {r!r}') print(f'Body: {r.content.decode()}') diff --git a/src/eddn/Gateway.py b/src/eddn/Gateway.py index ee360f1..d9e7dfd 100644 --- a/src/eddn/Gateway.py +++ b/src/eddn/Gateway.py @@ -166,23 +166,13 @@ def get_decompressed_message(): else: logger.debug('Content-Encoding indicates *not* compressed...') - form_enc_parsed = urlparse.parse_qs(request.body.read()) - if form_enc_parsed: - logger.debug('Request is form-encoded') - - # Uncompressed request. Bottle handles all of the parsing of the - # POST key/vals, or un-encoded body. - data_key = request.forms.get('data') - if data_key: - logger.debug('form-encoded POST request detected...') - # This is a form-encoded POST. Support the silly people. - message_body = data_key - - else: - raise MalformedUploadError( - "No 'data' POST key/value found. Check your POST key " - "name for spelling, and make sure you're passing a value." - ) + # Uncompressed request. Bottle handles all of the parsing of the + # POST key/vals, or un-encoded body. + data_key = request.forms.get('data') + if data_key: + logger.debug('form-encoded POST request detected...') + # This is a form-encoded POST. Support the silly people. + message_body = data_key else: logger.debug('Plain POST request detected...')