693 Commits

Author SHA1 Message Date
Athanasius
1a86e5242c Merge branch 'master' of ../../dev/EDDN Release/d.20220111-00 2022-01-11 12:51:03 +00:00
Athanasius
deb7383e32 docs/schemas: Additions and tweaks for uploaders
* Document the "you tried to use plain HTTP" response from Reverse
Proxy in the appropriate section.

* Switch URL for Gateway.py to `live` branch.

* 'disallowed' Schema Validation error cites **value**, not key *name*.

* Call out the possible need to explicitly subscribe to the empty topic
in order to receive messages... and that currently server-side topic
filtering isn't possible.
2022-01-11 12:48:40 +00:00
Athanasius
fb81c51a12 Merge branch 'master' of ../../dev/EDDN 2022-01-11 12:31:20 +00:00
Athanasius
1943003b06
README: Complete re-write
* Expand the initial blurb into a full 'About' section.
* Making 'Using EDDN' an actual section, to facilitate migrating away
  from the wiki page.
* Guide players towards some useful information (guide and sites).
* Ensure developers know to check the *live* documentation.
* Explicitly link to the current wiki page and EDDN status.
* Add 'Hosting of the live service' section, and only list the current
  situation (Vivio hosting is 4-5 years ago now, and Anthor's was never
  called out).
2022-01-11 12:24:33 +00:00
Athanasius
084ea35286 setup.py: Change "live branch for live" check to use setup_env.py
The whole build/install process is dependent on the EDDN_ENV value
anyway, so we might as well use it.

This way if a future setup doesn't have a specific string in CWD this
check doesn't break.

Yes, this whole check means you MUST be building from a git checkout,
not the files sourced by some other means.
2022-01-10 16:23:14 +00:00
Athanasius
35c4f89c68 setup.py: Detect if not using live branch in live directory 2022-01-10 16:17:39 +00:00
Athanasius
495e27861e scripts: Make it obvious that test-bad-gzip.py doesn't work 2022-01-10 15:53:44 +00:00
Athanasius
a8098f45b9 Move eddn-report-log-errors into scripts/ 2022-01-10 09:42:17 +00:00
Athanasius
aa7fabf134 Merge branch 'master' into develop 2022-01-10 09:40:14 +00:00
Athanasius
66d39a1bba Merge branch 'master' into develop 2022-01-10 09:38:53 +00:00
Athanasius
f31974d6ff
Merge pull request #162 from EDCD/enhancement/161/gateway-improve-misc-error-reports
Gateway: Properly report all errors to uploaders
2022-01-09 16:41:15 +00:00
Athanasius
ddb9480ee6
Gateway/testscripts: Add a small README 2022-01-09 16:27:08 +00:00
Athanasius
2a8eb8d021
Gateway: Add test scripts and supporting files
These are what I was using on my home server to test the prior code
changes to how the Gateway code reports errors.

Ultimately these should become a part of proper tests, but for now
they're at least in the repository for anyone to utilise.
2022-01-09 16:21:23 +00:00
Athanasius
75c117fb12
docs/schemas: form-encoded messages are deprecated and not documented
At some point I might add logging to see if any sender even *is*
utilising this, and if not actually remove the possibility entirely.
2022-01-09 16:14:00 +00:00
Athanasius
11d3501c59
Gateway/errors: More documentation tweaking 2022-01-09 16:13:01 +00:00
Athanasius
4ad1519d1d
Gateway/errors: Improve general outline of documentation section 2022-01-09 16:10:01 +00:00
Athanasius
ccde820ba7
Gateway/errors: 'Schema Validation' properly tagged and documented 2022-01-09 16:02:28 +00:00
Athanasius
e111fb8415
Gateway/errors: Outdate Schema: Tweak message and document 2022-01-09 15:48:27 +00:00
Athanasius
e931bfff96
Gateway/errors: MalformedUploadError isn't raised upon JSON issue 2022-01-09 15:45:56 +00:00
Athanasius
b9b59329d5
Gateway/errors: Document 'Malformed Upload' error 2022-01-09 15:43:51 +00:00
Athanasius
4da60215f0
Gateway/errors: Tag FAIL if a zlib.error 2022-01-09 15:43:12 +00:00
Athanasius
8edae919e2
Gateway/errors: Call out if FAIL is specifically on JSON parsing 2022-01-09 15:38:29 +00:00
Athanasius
35cd3c3294
Merge branch 'enhancement/161/gateway-improve-misc-error-reports' of github.com:EDSM-NET/EDDN into enhancement/161/gateway-improve-misc-error-reports 2022-01-09 15:33:24 +00:00
Athanasius
81a70572c9 Gateway: Properly report 'not compressed, badly form-encoded' to uploaders
* This code worked if the request was *properly* form-encoded, with a
'data' key whose value was a valid message.

* It failed to detect where the request was form-encoded, with without a
'data' key.  It would just assume 'not form-encoded' in that case, then
fail later on JSON parsing.

Thus, re-use the `urlparse.parse_qs()` check for form-encoded format.
This passes:

  1. Properly, `data` key, form-encoded with valid value is fully JSON
  parsed, schema checked and accepted.
  2. *NOT* compressed *or* form-encoded valid message is properly parsed
  and accepted.
  2. Uncompressed, form-encoded, but no `data` key correctly returns the
  same error status and body as the compressed+form-encoded+no data key
  path.
2022-01-09 15:31:39 +00:00
Athanasius
faa2e25d62 Gateway: Properly report 'badly form encoded' to uploaders
This currently only applies if the data was also, validly, compressed.
2022-01-09 15:31:39 +00:00
Athanasius
613ef6deab Gateway: Properly report zlib.decompress() errors to uploaders
This also adds some debug/error logging to the code path.
2022-01-09 15:31:39 +00:00
Athanasius
e58dc3b5d2
Merge pull request #160 from EDCD/enhancement/document-gateway-responses
docs: Possible server responses, and handling them
2022-01-09 15:29:32 +00:00
Athanasius
c6a63c5a93
docs/schemas: Journal timestamps *are* trustworthy as UTC 2022-01-09 15:25:55 +00:00
Athanasius
44b5a1d789
docs/schemas: slight wording tweak about live schemas 2022-01-09 14:57:05 +00:00
Athanasius
36de2145d7
docs: schemas: General improvement pass to aid new developers
* Make lots of 'obvious' things explicit, e.g. HTTP 1.1, not HTTP/2, and
  HTTPS not plain HTTP.
* The live service should always be using the schemas as present in the
  live branch, not master or another branch.
* A 'good' message will receive 'HTTP 200' status *and* a body of `OK`.
2022-01-09 14:52:41 +00:00
Athanasius
424fa72557 setup.py: Add 'enforce correct branch' TODO 2022-01-09 13:19:40 +00:00
Athanasius
1bb8a37c34 scripts: Add eddn-report-log-errors
This looks at gateway.log files for any 'ERROR' lines.  The output
should be any lines representing an error that hasn't yet been reported
to the softwareName's developer.

NB: Absolutely relies on the developer changing the softwareVersion
after applying a fix.
2022-01-09 13:09:04 +00:00
Athanasius
10d70bfe77 Gateway: Properly report 'not compressed, badly form-encoded' to uploaders
* This code worked if the request was *properly* form-encoded, with a
'data' key whose value was a valid message.

* It failed to detect where the request was form-encoded, with without a
'data' key.  It would just assume 'not form-encoded' in that case, then
fail later on JSON parsing.

Thus, re-use the `urlparse.parse_qs()` check for form-encoded format.
This passes:

  1. Properly, `data` key, form-encoded with valid value is fully JSON
  parsed, schema checked and accepted.
  2. *NOT* compressed *or* form-encoded valid message is properly parsed
  and accepted.
  2. Uncompressed, form-encoded, but no `data` key correctly returns the
  same error status and body as the compressed+form-encoded+no data key
  path.
2022-01-07 16:45:37 +00:00
Athanasius
872af7f594 Gateway: Properly report 'badly form encoded' to uploaders
This currently only applies if the data was also, validly, compressed.
2022-01-07 16:32:10 +00:00
Athanasius
10b12cf74b Gateway: Properly report zlib.decompress() errors to uploaders
This also adds some debug/error logging to the code path.
2022-01-07 16:04:00 +00:00
Athanasius
207068f156
docs: Cite issue on "some error bodies don't have FAIL: prefix" 2022-01-07 15:37:07 +00:00
Athanasius
d2c4c98c2b
docs: Possible server responses, and handling them 2022-01-07 15:20:05 +00:00
Athanasius
685f6a4f0c Merge branch 'master' of github.com:EDCD/EDDN Release/d.20220107-00 2022-01-07 10:41:55 +00:00
Athanasius
0bf8fc3abb
Merge pull request #159 from EDCD/enhancement/158/request-size-limit
Address `bottle` default request size rejecting some messages
2022-01-07 10:41:21 +00:00
Athanasius
1134a6c9b4 Gateway: Only log first 512 characters of invalid JSON
Don't want to spam the logs with up to 1MiB per bad message.
2022-01-06 17:39:01 +00:00
Athanasius
9f219da6a6 Gateway: Expand on logging for interpretation of errors
This includes logging the *full* (uncompressed) message if it fails to
parse as JSON.
2022-01-06 17:36:39 +00:00
Athanasius
1371f71217 Gateway: Set up proper logger formatting
For some reason the milliseconds portion of the %S timestamp is using a
comma for decimals separator, despite 'locale' saying we're set to (US)
English.  /tableflip
2022-01-06 14:06:59 +00:00
Athanasius
377bdd3833 Gateway: Add logging per Accepted request
Actually some logging was already there, just the logger had never been
set up properly, but then I decided to make the format of this message
more useful.
2022-01-06 13:54:33 +00:00
Athanasius
0e80c76cb5 Gateway: Set bottle request limit to 1MiB 2022-01-06 13:07:53 +00:00
Athanasius
fa118d0183
schemas/codexentry: Clarify that what unset status_body_name means 2021-12-21 09:37:08 +00:00
Athanasius
f8c1ce7a3e Script for testing schemas 2021-11-07 16:22:14 +00:00
Athanasius
229ae1556a Merge branch 'develop' of github.com:EDCD/EDDN into develop 2021-11-04 17:15:17 +00:00
Athanasius
b78fcf349f
Merge branch 'develop' 2021-11-04 11:26:35 +00:00
Athanasius
ac0a872d30
Merge pull request #151 from EDCD/enhancement/document-all-schemas
Document all of the schemas in their own README
2021-11-04 11:26:04 +00:00
Athanasius
d6a43c04d9
schemas: README: Add 'Receiving messages' section 2021-11-04 11:14:47 +00:00