246 Commits

Author SHA1 Message Date
Athanasius
dd7e12fa28
Version: Bump to 2.0-alph0 for this python3 work 2022-08-18 15:18:00 +01:00
Athanasius
3b5d16c674
Relay: Full flake8/mypy pass 2022-08-18 15:17:59 +01:00
Athanasius
06e9442bea
Monitor: Make EnableCors.apply() static 2022-08-18 15:17:59 +01:00
Athanasius
07e332ff61
Monitor: types pass 2022-08-18 15:17:59 +01:00
Athanasius
33a09ef8b5
Monitor: docstring'd 2022-08-18 15:17:59 +01:00
Athanasius
b37307e6a2
Monitor: Initial flake8 pass 2022-08-18 15:17:58 +01:00
Athanasius
3b3e1a91c5
Gateway: Make the CORS apply() static
Quietens a PyCharm warning
2022-08-18 15:17:58 +01:00
Athanasius
99679d9d5b
Gateway: Minor renames to make PyCharm not gripe
These are related to local variables shadowing globals.
2022-08-18 15:17:58 +01:00
Athanasius
c11fa91162
Gateway: typing: main() 2022-08-18 15:17:58 +01:00
Athanasius
179dd8aebf
Gateway: typing: stats() 2022-08-18 15:17:58 +01:00
Athanasius
8825526a1c
Gateway: typing: health_check() 2022-08-18 15:17:57 +01:00
Athanasius
5939e7c889
Gateway: typing: upload() 2022-08-18 15:17:57 +01:00
Athanasius
9be3cd82d8
Gateway: typing: parse_and_error_handle() 2022-08-18 15:17:57 +01:00
Athanasius
35b90de06d
Gateway: typing: get_decompressed_message()
# Conflicts:
#	src/eddn/Gateway.py
2022-08-18 15:17:54 +01:00
Athanasius
74de8a07ee
Gateway: Start adding types/checking
# Conflicts:
#	src/eddn/Gateway.py
2022-08-18 15:17:29 +01:00
Athanasius
24c80c3132
Gateway: misc formatting pass 2022-08-18 14:45:00 +01:00
Athanasius
785378a9bc
Gateway: % -> f-string pass 2022-08-18 14:45:00 +01:00
Athanasius
048e908e09
Gateway: docstring pass 2022-08-18 14:45:00 +01:00
Athanasius
3b4fde42db
Gateway: snake_case variables 2022-08-18 14:44:59 +01:00
Athanasius
123769f8d4
Gateway: More import fixup 2022-08-18 14:44:59 +01:00
Athanasius
bf432a9708
Gateway: Remove un-used bottle imports 2022-08-18 14:44:59 +01:00
Athanasius
b127c63887
Gateway: Proper import order 2022-08-18 14:44:49 +01:00
Athanasius
ca3417c9c0
Gateway: Better top-level docstring 2022-08-18 14:43:16 +01:00
Athanasius
ff83ede948
Gateway: Remove all form-encoded support
This causes issues, at the least, with compressed messages that 'look' like
they decompressed body is form-encoded.  18385 messages in the last month
rejected due to this.

No actually valid form-encoded messages in that time frame.
2022-06-16 13:27:11 +01:00
Athanasius
56b4e20238
Settings.py: Add missing / for fsssignaldiscovered/1 schema 2022-06-15 14:03:02 +01:00
912fa0e064
WIP: FSSSignalDiscovered event support 2022-06-13 22:35:58 +03:00
Gareth Harper
b9e624a351 make gateway use new schema 2022-05-25 14:39:55 +00:00
Athanasius
8db880f95d schemas/FSSAllBodiesFound: Don't list in outdated schemas 2022-01-28 17:29:11 +00:00
robbyxp1
d4e7409ae0 Add to settings.py the fssallbodiesfound schemas 2022-01-28 17:29:11 +00:00
robbyxp1
3b7c002b51 Add schema to settings 2022-01-28 08:14:45 +00:00
Athanasius
8d5b03915b Gateway: Log where upload data is form-encoded
We suspect no-one is actually using this possible form of upload, and
ideally would remove the code for it entirely.  First we need to log any
uploaders that do use this form to be sure.
2022-01-18 15:53:06 +00:00
Athanasius
8f7910a5d9 Bouncer: Support --loglevel <level> CL arg
This also aligns the general logger setup with the other scripts
2022-01-18 15:22:02 +00:00
Athanasius
6986af7ca0 Settings.py: Remove import argparse
CL arg parsing is done in main scripts now.
2022-01-18 15:17:45 +00:00
Athanasius
fe214583aa Relay: Support --loglevel <level> fully
There's no real logging in Relay yet, but this makes it support
switching level anyway.
2022-01-18 15:16:32 +00:00
Athanasius
86b283f920 Monitor.py: Minimal changes to allow --loglevel <level>
The CL arg won't actually *do* anything yet, because this script doesn't
actually have a logger set up at all.  But we need to fake this else
eddn.conf.Settings.loadConfig() won't be called correctly or work.
2022-01-18 15:09:56 +00:00
Athanasius
fb3c0348ad Gateway: Support --loglevel <level> CL arg
NB: This required changes to src/eddn/conf/Settings.py as well, which
means other scripts, e.g. Monitor.py, will also need changing.
2022-01-18 15:02:38 +00:00
Athanasius
abcf472197 Gateway: Revert non-gzip form encoded check
Fixes the regression identified in #165

However the gzip code path can still erroneously think a decompressed
request body is form encoded when it is not.  This happens when any text
in the decompressed body matches the regex:

        .+=[^\&;]*

i.e. some text, followed by `=`, and then some more text, possibly
empty, followed by an ampersand `&`, or semi-colon `;`, or the end of the
string.

`&` and `;` are used to separate key=value pairs in form encoding, the
`=` separates a key from its value.
2022-01-11 15:06:02 +00:00
Athanasius
ccde820ba7
Gateway/errors: 'Schema Validation' properly tagged and documented 2022-01-09 16:02:28 +00:00
Athanasius
e111fb8415
Gateway/errors: Outdate Schema: Tweak message and document 2022-01-09 15:48:27 +00:00
Athanasius
e931bfff96
Gateway/errors: MalformedUploadError isn't raised upon JSON issue 2022-01-09 15:45:56 +00:00
Athanasius
4da60215f0
Gateway/errors: Tag FAIL if a zlib.error 2022-01-09 15:43:12 +00:00
Athanasius
8edae919e2
Gateway/errors: Call out if FAIL is specifically on JSON parsing 2022-01-09 15:38:29 +00:00
Athanasius
10d70bfe77 Gateway: Properly report 'not compressed, badly form-encoded' to uploaders
* This code worked if the request was *properly* form-encoded, with a
'data' key whose value was a valid message.

* It failed to detect where the request was form-encoded, with without a
'data' key.  It would just assume 'not form-encoded' in that case, then
fail later on JSON parsing.

Thus, re-use the `urlparse.parse_qs()` check for form-encoded format.
This passes:

  1. Properly, `data` key, form-encoded with valid value is fully JSON
  parsed, schema checked and accepted.
  2. *NOT* compressed *or* form-encoded valid message is properly parsed
  and accepted.
  2. Uncompressed, form-encoded, but no `data` key correctly returns the
  same error status and body as the compressed+form-encoded+no data key
  path.
2022-01-07 16:45:37 +00:00
Athanasius
872af7f594 Gateway: Properly report 'badly form encoded' to uploaders
This currently only applies if the data was also, validly, compressed.
2022-01-07 16:32:10 +00:00
Athanasius
10b12cf74b Gateway: Properly report zlib.decompress() errors to uploaders
This also adds some debug/error logging to the code path.
2022-01-07 16:04:00 +00:00
Athanasius
1134a6c9b4 Gateway: Only log first 512 characters of invalid JSON
Don't want to spam the logs with up to 1MiB per bad message.
2022-01-06 17:39:01 +00:00
Athanasius
9f219da6a6 Gateway: Expand on logging for interpretation of errors
This includes logging the *full* (uncompressed) message if it fails to
parse as JSON.
2022-01-06 17:36:39 +00:00
Athanasius
1371f71217 Gateway: Set up proper logger formatting
For some reason the milliseconds portion of the %S timestamp is using a
comma for decimals separator, despite 'locale' saying we're set to (US)
English.  /tableflip
2022-01-06 14:06:59 +00:00
Athanasius
377bdd3833 Gateway: Add logging per Accepted request
Actually some logging was already there, just the logger had never been
set up properly, but then I decided to make the format of this message
more useful.
2022-01-06 13:54:33 +00:00
Athanasius
0e80c76cb5 Gateway: Set bottle request limit to 1MiB 2022-01-06 13:07:53 +00:00