127 Commits

Author SHA1 Message Date
Athanasius
af9089ccee
Tweak Gateway, Monitor, Relay startup logging 2022-08-18 16:19:27 +01:00
Athanasius
1f991e7c44
Gateway: Remove un-necessary import urlparse 2022-08-18 16:13:59 +01:00
Athanasius
1384241f13
Fix "are you running this correctly?" check to utilise pathlib
As it was this only worked on systems with `/`-delimited path components.

So use `pathlib.Path.as_posix()` to ensure it *is* in that format.
2022-08-18 15:20:22 +01:00
Athanasius
6a4bad49c8
Gateway: Add 'are we running this correctly?' check
Also some misc import order fixups
2022-08-18 15:20:20 +01:00
Athanasius
b437df5af8
Move main scripts back into src/eddn/
So, yes, running any of these absolutely directly:

    cd src
    python eddn/Gateway.py

will fail.  But you *can*:

    cd src
    python -m eddn.Gateway

and they work perfectly fine.
2022-08-18 15:20:19 +01:00
Athanasius
c2dc30c8a2
Finally got the correct setup.py configuration for source of egg running
* py_modules parameter to setup() isn't documented in the setuptools
  docs, but is in a general 'python packaging' one.
* So now the main scripts are NOT within the `eddn` package..
* But all other code is...
* But the schema files don't need to be.

# Conflicts:
#	src/schemas/fssbodysignals-README.md
#	src/schemas/fssbodysignals-v1.0.json
#	src/schemas/fsssignaldiscovered-README.md
#	src/schemas/fsssignaldiscovered-v1.0.json
2022-08-18 15:20:18 +01:00
Athanasius
47ae96ddc1
More file re-arrangement, concentrating on running from source
With `import eddn.core.Validator` and the like it was actually picking
up the egg versions of the files, not local ones.

Currently this will run as per:

cd src/eddn
python Gateway.py --config ~/.local/share/eddn/dev/config.json

# Conflicts:
#	src/eddn/schemas/fssbodysignals-README.md
#	src/eddn/schemas/fssbodysignals-v1.0.json
#	src/eddn/schemas/fsssignaldiscovered-README.md
#	src/eddn/schemas/fsssignaldiscovered-v1.0.json
2022-08-18 15:20:18 +01:00
Athanasius
d36e45873d
Initial re-arrangement of files
* Move the service scripts out of src/eddn/ into src/
* Move the schemas/ to src/schemas/
* Some updates to setup.py, i.e. source of files, but the setuptools
  part isn't yet work.

# Conflicts:
#	src/schemas/fssbodysignals-README.md
#	src/schemas/fssbodysignals-v1.0.json
#	src/schemas/fsssignaldiscovered-README.md
#	src/schemas/fsssignaldiscovered-v1.0.json
2022-08-18 15:20:17 +01:00
Athanasius
1ae349a4b7
Make per-component logging more obvious 2022-08-18 15:20:15 +01:00
Athanasius
d79f4a5aa1
Monitor: Change to be able to run TLS-less
NB: Not yet changed the actual monitor web page files.
2022-08-18 15:20:15 +01:00
Athanasius
128dab965c
TLS: Optionally use TLS if you set non-empty CERT_FILE and KEY_FILE
Whilst we do want to go TLS-less in the actual EDDN code, for ease of
setting up automated end to end functional testing, leave the
possibility of running with TLS termination as well.
2022-08-18 15:20:15 +01:00
Athanasius
83e2b548ad
Gateway: Use a dict for kwargs to app.run()
This is so we can now adjust if we're putting the TLS cert/key files in.
2022-08-18 15:20:14 +01:00
Athanasius
d5dbc3262f
Gateway: Put remote_addr in [], as it could be multiple, comma-separated
# Conflicts:
#	src/eddn/Gateway.py

# Conflicts:
#	src/eddn/Gateway.py
2022-08-18 15:20:12 +01:00
Athanasius
8cf0d12d97
Move EDDNWSGIHandler into its own file 2022-08-18 15:19:58 +01:00
Athanasius
402758f1d6
Gateway: Fixed logging to be consistent and use client IP
* Send all the bottle server output through our logger.
* Ensure gevent uses client IP, not 127.0.0.1.
2022-08-18 15:19:58 +01:00
Athanasius
4b0898122d
Gateway: Don't use TLS cert in app setup
* The /upload/ functionality continues to work.
2022-08-18 15:19:57 +01:00
Athanasius
71ab6ba6be
Gateway: Remove extraneous noqa's 2022-08-18 15:19:56 +01:00
Athanasius
5d3358417d
Gateway: Format with black
# Conflicts:
#	src/eddn/Gateway.py
2022-08-18 15:19:50 +01:00
Athanasius
841c7e979e
Gateway: Use "" for strings throughout
# Conflicts:
#	src/eddn/Gateway.py

# Conflicts:
#	src/eddn/Gateway.py
2022-08-18 15:19:33 +01:00
Athanasius
241bd911dd
Gateway: flake8 and mypy pass
# Conflicts:
#	src/eddn/Gateway.py
2022-08-18 15:19:14 +01:00
Athanasius
3a55f6b005
Gateway: zmq's PUB is *only* in the main module now
But we still need to `import zmq.green as zmq` so use:

```python
from zmq import PUB as ZMQ_PUB
```
2022-08-18 15:18:53 +01:00
Athanasius
3dba22b6cb
Gateway: Remove "remember to do the python3 rebase" line 2022-08-18 15:18:52 +01:00
Athanasius
2b39934fd7
'Guard' non-code to ensure this gets updated for bottle size limit etc 2022-08-18 15:18:52 +01:00
Athanasius
5e51c604a4
flake8: minor cleanups
# Conflicts:
#	src/eddn/Gateway.py
2022-08-18 15:18:48 +01:00
Athanasius
529725d2db
Gateway -> Monitor now working
We literally weren't making use of the 'topic' in the message. Thus, so
as to avoid issues with trying to mash a string topic together with a
bytes (compressed) message, I've ripped that out.
2022-08-18 15:18:29 +01:00
Athanasius
fdee2c054f
python3: Gateway now working
Now I'm actually trying to run the code I'm finding more that needs
changing in order to run under python3
2022-08-18 15:18:28 +01:00
Athanasius
bd5784034a
Validator: Full flake8 and mypy pass 2022-08-18 15:18:28 +01:00
Athanasius
e6529060e2
StatsCollector: Full flake8 and mypy pass
A function name refactor touches other files.
2022-08-18 15:18:27 +01:00
Athanasius
e9b4afcdf6
Use that bottle Plugin.apply() signature throughout
# Conflicts:
#	src/eddn/Gateway.py
2022-08-18 15:18:24 +01:00
Athanasius
fcfe9e01b2
Settings.loadConfig() got renamed to load_config() 2022-08-18 15:18:00 +01:00
Athanasius
3b3e1a91c5
Gateway: Make the CORS apply() static
Quietens a PyCharm warning
2022-08-18 15:17:58 +01:00
Athanasius
99679d9d5b
Gateway: Minor renames to make PyCharm not gripe
These are related to local variables shadowing globals.
2022-08-18 15:17:58 +01:00
Athanasius
c11fa91162
Gateway: typing: main() 2022-08-18 15:17:58 +01:00
Athanasius
179dd8aebf
Gateway: typing: stats() 2022-08-18 15:17:58 +01:00
Athanasius
8825526a1c
Gateway: typing: health_check() 2022-08-18 15:17:57 +01:00
Athanasius
5939e7c889
Gateway: typing: upload() 2022-08-18 15:17:57 +01:00
Athanasius
9be3cd82d8
Gateway: typing: parse_and_error_handle() 2022-08-18 15:17:57 +01:00
Athanasius
35b90de06d
Gateway: typing: get_decompressed_message()
# Conflicts:
#	src/eddn/Gateway.py
2022-08-18 15:17:54 +01:00
Athanasius
74de8a07ee
Gateway: Start adding types/checking
# Conflicts:
#	src/eddn/Gateway.py
2022-08-18 15:17:29 +01:00
Athanasius
24c80c3132
Gateway: misc formatting pass 2022-08-18 14:45:00 +01:00
Athanasius
785378a9bc
Gateway: % -> f-string pass 2022-08-18 14:45:00 +01:00
Athanasius
048e908e09
Gateway: docstring pass 2022-08-18 14:45:00 +01:00
Athanasius
3b4fde42db
Gateway: snake_case variables 2022-08-18 14:44:59 +01:00
Athanasius
123769f8d4
Gateway: More import fixup 2022-08-18 14:44:59 +01:00
Athanasius
bf432a9708
Gateway: Remove un-used bottle imports 2022-08-18 14:44:59 +01:00
Athanasius
b127c63887
Gateway: Proper import order 2022-08-18 14:44:49 +01:00
Athanasius
ca3417c9c0
Gateway: Better top-level docstring 2022-08-18 14:43:16 +01:00
Athanasius
ff83ede948
Gateway: Remove all form-encoded support
This causes issues, at the least, with compressed messages that 'look' like
they decompressed body is form-encoded.  18385 messages in the last month
rejected due to this.

No actually valid form-encoded messages in that time frame.
2022-06-16 13:27:11 +01:00
Athanasius
8d5b03915b Gateway: Log where upload data is form-encoded
We suspect no-one is actually using this possible form of upload, and
ideally would remove the code for it entirely.  First we need to log any
uploaders that do use this form to be sure.
2022-01-18 15:53:06 +00:00
Athanasius
fb3c0348ad Gateway: Support --loglevel <level> CL arg
NB: This required changes to src/eddn/conf/Settings.py as well, which
means other scripts, e.g. Monitor.py, will also need changing.
2022-01-18 15:02:38 +00:00