705 Commits

Author SHA1 Message Date
Athanasius
1ee25a69d6 Merge branch 'master' of github.com:EDCD/EDDN 2022-01-11 15:42:22 +00:00
Athanasius
a4eb7548d9
Merge pull request #166 from EDCD/fix/165/bad-formencoded-detection
Gateway: Revert non-gzip form encoded check
2022-01-11 15:39:01 +00:00
Athanasius
6799a7fae9 scripts/testing: Target dev:4432, not beta:4431
In the long-term these all need to support a CL arg to select the
target.
2022-01-11 15:36:53 +00:00
Athanasius
a1d347aec5 scripts/testing: Add general "uses gzip" script 2022-01-11 15:35:24 +00:00
Athanasius
abcf472197 Gateway: Revert non-gzip form encoded check
Fixes the regression identified in #165

However the gzip code path can still erroneously think a decompressed
request body is form encoded when it is not.  This happens when any text
in the decompressed body matches the regex:

        .+=[^\&;]*

i.e. some text, followed by `=`, and then some more text, possibly
empty, followed by an ampersand `&`, or semi-colon `;`, or the end of the
string.

`&` and `;` are used to separate key=value pairs in form encoding, the
`=` separates a key from its value.
2022-01-11 15:06:02 +00:00
Athanasius
6add1d5c6b Merge branch 'master' of github.com:EDCD/EDDN 2022-01-11 15:01:20 +00:00
Athanasius
759b14c844 scripts/testing: Add "valid JSON looks like form-encoded" file
Ref: https://github.com/EDCD/EDDN/issues/165
2022-01-11 14:40:16 +00:00
Athanasius
2d60092d0e
README: Emphasise developers must comply with documentation 2022-01-11 13:47:41 +00:00
Athanasius
1caf5f0600
README: 'Documentation' section removed as un-necessary 2022-01-11 13:44:42 +00:00
Athanasius
01ce718ab8
Merge branch 'master' of github.com:EDSM-NET/EDDN 2022-01-11 13:42:48 +00:00
Athanasius
40262322ff
README: Expansion and tweaks for replacing the Wiki page
* A few uploading PC softwares are listed to lower the barrier for a new
  player finding such.  This is *never* going to be an exhaustive list.
* Added section listing console options for contributing data.  Again,
  not intended to be exhaustive.
* Layout/sections tweaked.  Hopefully the use of double horizontal rules
  emphasises which section text is a part of.
2022-01-11 13:40:42 +00:00
Athanasius
a39f24c55d
README: Add 'Contact' section at the bottom 2022-01-11 13:13:06 +00:00
Athanasius
1a86e5242c Merge branch 'master' of ../../dev/EDDN Release/d.20220111-00 2022-01-11 12:51:03 +00:00
Athanasius
deb7383e32 docs/schemas: Additions and tweaks for uploaders
* Document the "you tried to use plain HTTP" response from Reverse
Proxy in the appropriate section.

* Switch URL for Gateway.py to `live` branch.

* 'disallowed' Schema Validation error cites **value**, not key *name*.

* Call out the possible need to explicitly subscribe to the empty topic
in order to receive messages... and that currently server-side topic
filtering isn't possible.
2022-01-11 12:48:40 +00:00
Athanasius
fb81c51a12 Merge branch 'master' of ../../dev/EDDN 2022-01-11 12:31:20 +00:00
Athanasius
1943003b06
README: Complete re-write
* Expand the initial blurb into a full 'About' section.
* Making 'Using EDDN' an actual section, to facilitate migrating away
  from the wiki page.
* Guide players towards some useful information (guide and sites).
* Ensure developers know to check the *live* documentation.
* Explicitly link to the current wiki page and EDDN status.
* Add 'Hosting of the live service' section, and only list the current
  situation (Vivio hosting is 4-5 years ago now, and Anthor's was never
  called out).
2022-01-11 12:24:33 +00:00
Athanasius
084ea35286 setup.py: Change "live branch for live" check to use setup_env.py
The whole build/install process is dependent on the EDDN_ENV value
anyway, so we might as well use it.

This way if a future setup doesn't have a specific string in CWD this
check doesn't break.

Yes, this whole check means you MUST be building from a git checkout,
not the files sourced by some other means.
2022-01-10 16:23:14 +00:00
Athanasius
35c4f89c68 setup.py: Detect if not using live branch in live directory 2022-01-10 16:17:39 +00:00
Athanasius
495e27861e scripts: Make it obvious that test-bad-gzip.py doesn't work 2022-01-10 15:53:44 +00:00
Athanasius
a8098f45b9 Move eddn-report-log-errors into scripts/ 2022-01-10 09:42:17 +00:00
Athanasius
aa7fabf134 Merge branch 'master' into develop 2022-01-10 09:40:14 +00:00
Athanasius
66d39a1bba Merge branch 'master' into develop 2022-01-10 09:38:53 +00:00
Athanasius
f31974d6ff
Merge pull request #162 from EDCD/enhancement/161/gateway-improve-misc-error-reports
Gateway: Properly report all errors to uploaders
2022-01-09 16:41:15 +00:00
Athanasius
ddb9480ee6
Gateway/testscripts: Add a small README 2022-01-09 16:27:08 +00:00
Athanasius
2a8eb8d021
Gateway: Add test scripts and supporting files
These are what I was using on my home server to test the prior code
changes to how the Gateway code reports errors.

Ultimately these should become a part of proper tests, but for now
they're at least in the repository for anyone to utilise.
2022-01-09 16:21:23 +00:00
Athanasius
75c117fb12
docs/schemas: form-encoded messages are deprecated and not documented
At some point I might add logging to see if any sender even *is*
utilising this, and if not actually remove the possibility entirely.
2022-01-09 16:14:00 +00:00
Athanasius
11d3501c59
Gateway/errors: More documentation tweaking 2022-01-09 16:13:01 +00:00
Athanasius
4ad1519d1d
Gateway/errors: Improve general outline of documentation section 2022-01-09 16:10:01 +00:00
Athanasius
ccde820ba7
Gateway/errors: 'Schema Validation' properly tagged and documented 2022-01-09 16:02:28 +00:00
Athanasius
e111fb8415
Gateway/errors: Outdate Schema: Tweak message and document 2022-01-09 15:48:27 +00:00
Athanasius
e931bfff96
Gateway/errors: MalformedUploadError isn't raised upon JSON issue 2022-01-09 15:45:56 +00:00
Athanasius
b9b59329d5
Gateway/errors: Document 'Malformed Upload' error 2022-01-09 15:43:51 +00:00
Athanasius
4da60215f0
Gateway/errors: Tag FAIL if a zlib.error 2022-01-09 15:43:12 +00:00
Athanasius
8edae919e2
Gateway/errors: Call out if FAIL is specifically on JSON parsing 2022-01-09 15:38:29 +00:00
Athanasius
35cd3c3294
Merge branch 'enhancement/161/gateway-improve-misc-error-reports' of github.com:EDSM-NET/EDDN into enhancement/161/gateway-improve-misc-error-reports 2022-01-09 15:33:24 +00:00
Athanasius
81a70572c9 Gateway: Properly report 'not compressed, badly form-encoded' to uploaders
* This code worked if the request was *properly* form-encoded, with a
'data' key whose value was a valid message.

* It failed to detect where the request was form-encoded, with without a
'data' key.  It would just assume 'not form-encoded' in that case, then
fail later on JSON parsing.

Thus, re-use the `urlparse.parse_qs()` check for form-encoded format.
This passes:

  1. Properly, `data` key, form-encoded with valid value is fully JSON
  parsed, schema checked and accepted.
  2. *NOT* compressed *or* form-encoded valid message is properly parsed
  and accepted.
  2. Uncompressed, form-encoded, but no `data` key correctly returns the
  same error status and body as the compressed+form-encoded+no data key
  path.
2022-01-09 15:31:39 +00:00
Athanasius
faa2e25d62 Gateway: Properly report 'badly form encoded' to uploaders
This currently only applies if the data was also, validly, compressed.
2022-01-09 15:31:39 +00:00
Athanasius
613ef6deab Gateway: Properly report zlib.decompress() errors to uploaders
This also adds some debug/error logging to the code path.
2022-01-09 15:31:39 +00:00
Athanasius
e58dc3b5d2
Merge pull request #160 from EDCD/enhancement/document-gateway-responses
docs: Possible server responses, and handling them
2022-01-09 15:29:32 +00:00
Athanasius
c6a63c5a93
docs/schemas: Journal timestamps *are* trustworthy as UTC 2022-01-09 15:25:55 +00:00
Athanasius
44b5a1d789
docs/schemas: slight wording tweak about live schemas 2022-01-09 14:57:05 +00:00
Athanasius
36de2145d7
docs: schemas: General improvement pass to aid new developers
* Make lots of 'obvious' things explicit, e.g. HTTP 1.1, not HTTP/2, and
  HTTPS not plain HTTP.
* The live service should always be using the schemas as present in the
  live branch, not master or another branch.
* A 'good' message will receive 'HTTP 200' status *and* a body of `OK`.
2022-01-09 14:52:41 +00:00
Athanasius
424fa72557 setup.py: Add 'enforce correct branch' TODO 2022-01-09 13:19:40 +00:00
Athanasius
1bb8a37c34 scripts: Add eddn-report-log-errors
This looks at gateway.log files for any 'ERROR' lines.  The output
should be any lines representing an error that hasn't yet been reported
to the softwareName's developer.

NB: Absolutely relies on the developer changing the softwareVersion
after applying a fix.
2022-01-09 13:09:04 +00:00
Athanasius
10d70bfe77 Gateway: Properly report 'not compressed, badly form-encoded' to uploaders
* This code worked if the request was *properly* form-encoded, with a
'data' key whose value was a valid message.

* It failed to detect where the request was form-encoded, with without a
'data' key.  It would just assume 'not form-encoded' in that case, then
fail later on JSON parsing.

Thus, re-use the `urlparse.parse_qs()` check for form-encoded format.
This passes:

  1. Properly, `data` key, form-encoded with valid value is fully JSON
  parsed, schema checked and accepted.
  2. *NOT* compressed *or* form-encoded valid message is properly parsed
  and accepted.
  2. Uncompressed, form-encoded, but no `data` key correctly returns the
  same error status and body as the compressed+form-encoded+no data key
  path.
2022-01-07 16:45:37 +00:00
Athanasius
872af7f594 Gateway: Properly report 'badly form encoded' to uploaders
This currently only applies if the data was also, validly, compressed.
2022-01-07 16:32:10 +00:00
Athanasius
10b12cf74b Gateway: Properly report zlib.decompress() errors to uploaders
This also adds some debug/error logging to the code path.
2022-01-07 16:04:00 +00:00
Athanasius
207068f156
docs: Cite issue on "some error bodies don't have FAIL: prefix" 2022-01-07 15:37:07 +00:00
Athanasius
d2c4c98c2b
docs: Possible server responses, and handling them 2022-01-07 15:20:05 +00:00
Athanasius
685f6a4f0c Merge branch 'master' of github.com:EDCD/EDDN Release/d.20220107-00 2022-01-07 10:41:55 +00:00