third-party-mirrors/EDDN
1
0
Fork 0
mirror of https://github.com/EDCD/EDDN.git synced 2025-05-02 16:11:31 +03:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,025 Commits 9 Branches 12 Tags
Commit Graph

116 Commits

Author SHA1 Message Date
Athanasius
83e2b548ad
Gateway: Use a dict for kwargs to app.run() 
This is so we can now adjust if we're putting the TLS cert/key files in.
 2022-08-18 15:20:14 +01:00
Athanasius
d5dbc3262f
Gateway: Put remote_addr in [], as it could be multiple, comma-separated 
# Conflicts:
#	src/eddn/Gateway.py

# Conflicts:
#	src/eddn/Gateway.py
 2022-08-18 15:20:12 +01:00
Athanasius
8cf0d12d97
Move EDDNWSGIHandler into its own file 2022-08-18 15:19:58 +01:00
Athanasius
402758f1d6
Gateway: Fixed logging to be consistent and use client IP 
* Send all the bottle server output through our logger.
* Ensure gevent uses client IP, not 127.0.0.1.
 2022-08-18 15:19:58 +01:00
Athanasius
4b0898122d
Gateway: Don't use TLS cert in app setup 
* The /upload/ functionality continues to work.
 2022-08-18 15:19:57 +01:00
Athanasius
71ab6ba6be
Gateway: Remove extraneous noqa's 2022-08-18 15:19:56 +01:00
Athanasius
5d3358417d
Gateway: Format with black 
# Conflicts:
#	src/eddn/Gateway.py
 2022-08-18 15:19:50 +01:00
Athanasius
841c7e979e
Gateway: Use "" for strings throughout 
# Conflicts:
#	src/eddn/Gateway.py

# Conflicts:
#	src/eddn/Gateway.py
 2022-08-18 15:19:33 +01:00
Athanasius
241bd911dd
Gateway: flake8 and mypy pass 
# Conflicts:
#	src/eddn/Gateway.py
 2022-08-18 15:19:14 +01:00
Athanasius
3a55f6b005
Gateway: zmq's PUB is *only* in the main module now 
But we still need to `import zmq.green as zmq` so use:

```python
from zmq import PUB as ZMQ_PUB
```
 2022-08-18 15:18:53 +01:00
Athanasius
3dba22b6cb
Gateway: Remove "remember to do the python3 rebase" line 2022-08-18 15:18:52 +01:00
Athanasius
2b39934fd7
'Guard' non-code to ensure this gets updated for bottle size limit etc 2022-08-18 15:18:52 +01:00
Athanasius
5e51c604a4
flake8: minor cleanups 
# Conflicts:
#	src/eddn/Gateway.py
 2022-08-18 15:18:48 +01:00
Athanasius
529725d2db
Gateway -> Monitor now working 
We literally weren't making use of the 'topic' in the message. Thus, so
as to avoid issues with trying to mash a string topic together with a
bytes (compressed) message, I've ripped that out.
 2022-08-18 15:18:29 +01:00
Athanasius
fdee2c054f
python3: Gateway now working 
Now I'm actually trying to run the code I'm finding more that needs
changing in order to run under python3
 2022-08-18 15:18:28 +01:00
Athanasius
bd5784034a
Validator: Full flake8 and mypy pass 2022-08-18 15:18:28 +01:00
Athanasius
e6529060e2
StatsCollector: Full flake8 and mypy pass 
A function name refactor touches other files.
 2022-08-18 15:18:27 +01:00
Athanasius
e9b4afcdf6
Use that bottle Plugin.apply() signature throughout 
# Conflicts:
#	src/eddn/Gateway.py
 2022-08-18 15:18:24 +01:00
Athanasius
fcfe9e01b2
Settings.loadConfig() got renamed to load_config() 2022-08-18 15:18:00 +01:00
Athanasius
3b3e1a91c5
Gateway: Make the CORS apply() static 
Quietens a PyCharm warning
 2022-08-18 15:17:58 +01:00
Athanasius
99679d9d5b
Gateway: Minor renames to make PyCharm not gripe 
These are related to local variables shadowing globals.
 2022-08-18 15:17:58 +01:00
Athanasius
c11fa91162
Gateway: typing: main() 2022-08-18 15:17:58 +01:00
Athanasius
179dd8aebf
Gateway: typing: stats() 2022-08-18 15:17:58 +01:00
Athanasius
8825526a1c
Gateway: typing: health_check() 2022-08-18 15:17:57 +01:00
Athanasius
5939e7c889
Gateway: typing: upload() 2022-08-18 15:17:57 +01:00
Athanasius
9be3cd82d8
Gateway: typing: parse_and_error_handle() 2022-08-18 15:17:57 +01:00
Athanasius
35b90de06d
Gateway: typing: get_decompressed_message() 
# Conflicts:
#	src/eddn/Gateway.py
 2022-08-18 15:17:54 +01:00
Athanasius
74de8a07ee
Gateway: Start adding types/checking 
# Conflicts:
#	src/eddn/Gateway.py
 2022-08-18 15:17:29 +01:00
Athanasius
24c80c3132
Gateway: misc formatting pass 2022-08-18 14:45:00 +01:00
Athanasius
785378a9bc
Gateway: % -> f-string pass 2022-08-18 14:45:00 +01:00
Athanasius
048e908e09
Gateway: docstring pass 2022-08-18 14:45:00 +01:00
Athanasius
3b4fde42db
Gateway: snake_case variables 2022-08-18 14:44:59 +01:00
Athanasius
123769f8d4
Gateway: More import fixup 2022-08-18 14:44:59 +01:00
Athanasius
bf432a9708
Gateway: Remove un-used bottle imports 2022-08-18 14:44:59 +01:00
Athanasius
b127c63887
Gateway: Proper import order 2022-08-18 14:44:49 +01:00
Athanasius
ca3417c9c0
Gateway: Better top-level docstring 2022-08-18 14:43:16 +01:00
Athanasius
ff83ede948
Gateway: Remove all form-encoded support 
This causes issues, at the least, with compressed messages that 'look' like
they decompressed body is form-encoded.  18385 messages in the last month
rejected due to this.

No actually valid form-encoded messages in that time frame.
 2022-06-16 13:27:11 +01:00
Athanasius
8d5b03915b Gateway: Log where upload data is form-encoded 
We suspect no-one is actually using this possible form of upload, and
ideally would remove the code for it entirely.  First we need to log any
uploaders that do use this form to be sure.
 2022-01-18 15:53:06 +00:00
Athanasius
fb3c0348ad Gateway: Support --loglevel <level> CL arg 
NB: This required changes to src/eddn/conf/Settings.py as well, which
means other scripts, e.g. Monitor.py, will also need changing.
 2022-01-18 15:02:38 +00:00
Athanasius
abcf472197 Gateway: Revert non-gzip form encoded check 
Fixes the regression identified in #165

However the gzip code path can still erroneously think a decompressed
request body is form encoded when it is not.  This happens when any text
in the decompressed body matches the regex:

        .+=[^\&;]*

i.e. some text, followed by `=`, and then some more text, possibly
empty, followed by an ampersand `&`, or semi-colon `;`, or the end of the
string.

`&` and `;` are used to separate key=value pairs in form encoding, the
`=` separates a key from its value.
 2022-01-11 15:06:02 +00:00
Athanasius
ccde820ba7
Gateway/errors: 'Schema Validation' properly tagged and documented 2022-01-09 16:02:28 +00:00
Athanasius
e111fb8415
Gateway/errors: Outdate Schema: Tweak message and document 2022-01-09 15:48:27 +00:00
Athanasius
e931bfff96
Gateway/errors: MalformedUploadError isn't raised upon JSON issue 2022-01-09 15:45:56 +00:00
Athanasius
4da60215f0
Gateway/errors: Tag FAIL if a zlib.error 2022-01-09 15:43:12 +00:00
Athanasius
8edae919e2
Gateway/errors: Call out if FAIL is specifically on JSON parsing 2022-01-09 15:38:29 +00:00
Athanasius
10d70bfe77 Gateway: Properly report 'not compressed, badly form-encoded' to uploaders 
* This code worked if the request was *properly* form-encoded, with a
'data' key whose value was a valid message.

* It failed to detect where the request was form-encoded, with without a
'data' key.  It would just assume 'not form-encoded' in that case, then
fail later on JSON parsing.

Thus, re-use the `urlparse.parse_qs()` check for form-encoded format.
This passes:

  1. Properly, `data` key, form-encoded with valid value is fully JSON
  parsed, schema checked and accepted.
  2. *NOT* compressed *or* form-encoded valid message is properly parsed
  and accepted.
  2. Uncompressed, form-encoded, but no `data` key correctly returns the
  same error status and body as the compressed+form-encoded+no data key
  path.
 2022-01-07 16:45:37 +00:00
Athanasius
872af7f594 Gateway: Properly report 'badly form encoded' to uploaders 
This currently only applies if the data was also, validly, compressed.
 2022-01-07 16:32:10 +00:00
Athanasius
10b12cf74b Gateway: Properly report zlib.decompress() errors to uploaders 
This also adds some debug/error logging to the code path.
 2022-01-07 16:04:00 +00:00
Athanasius
1134a6c9b4 Gateway: Only log first 512 characters of invalid JSON 
Don't want to spam the logs with up to 1MiB per bad message.
 2022-01-06 17:39:01 +00:00
Athanasius
9f219da6a6 Gateway: Expand on logging for interpretation of errors 
This includes logging the *full* (uncompressed) message if it fails to
parse as JSON.
 2022-01-06 17:36:39 +00:00