Commit Graph
1073 Commits
Author SHA1 Message Date
Athanasius e28ab45be4 Gateway: Pass headers and body into get_decompressed_message()
This makes it easier to write unit tests for this function, not having to
somehow mock up the bottle request/body stuff.
2022-08-19 16:27:22 +01:00
Athanasius 580160b95b Relay: Add TRACE logging & use in key places
* Define our own TRACE logging level ('deeper' than DEBUG).
* Define `.trace()` method to use it.
* `logger.trace(...)` for duplicate or 'sent to Listeners' messages.

NB: mypy is still unhappy about Logger.trace, despite attempting the same
   workaround asin EDMC.  Hence the `  # type: ignore` on uses.
2022-08-19 16:25:30 +01:00
Athanasius 58c68ec17e tests/Gateway/post_upload: Fix assert
* dc_msg is a webtest object, so need to explicitly check its `body`.
* And that `body` is bytes, not str, so test against `b'OK'`.
2022-08-19 15:53:15 +01:00
Athanasius 00fed9b36a Gateway: Split some of main() into setup_bottle_app()
* This *should* facilitate some easier testing.
2022-08-19 15:13:33 +01:00
Athanasius 8a5bd46c33 tests/config: Gateway settings file
* Currently sets TLS cert/key files to `""` so as to be TLS-free.
2022-08-19 15:08:21 +01:00
Athanasius 6b339e3b11 tests/gateway/post_upload: Use webtest.TestApp for functional tests 2022-08-19 15:07:24 +01:00
Athanasius 30282ba95e tests/conftest: pytest top-level configuration, mostly Fixtures
* `eddn_message()` fixture, set up to return a method when used, such that
  *that* can be called with a key to look up the approproiate test message.
* `test_messages` dictionary to support that.
2022-08-19 15:01:43 +01:00
Athanasius 87c79a3427 tests: Gateway: parse_and_error_handle: Valid journal/scan message 2022-08-18 17:55:27 +01:00
Athanasius ba43199a5d tests: Gateway: parse_and_error_handle: No softwareName 2022-08-18 17:54:50 +01:00
Athanasius ddf9f3524f tests: Gateway: parse_and_error_handle: Two tests to get started
This is Athanasius feeling out how best to add tests to this ancient code.

* Test that eddn.Gateway.parse_and_error_handle() returns the correct error
  string for:
  - Invalid JSON being passed in.
  - An outdated schema being cited.
2022-08-18 17:23:08 +01:00
Athanasius 76251cfd30 Settings: Correct comment typos about reverse proxy setup 2022-08-18 16:20:51 +01:00
Athanasius af9089ccee Tweak Gateway, Monitor, Relay startup logging 2022-08-18 16:19:27 +01:00
Athanasius 5365d5efd6 Bouncer: Remove defunct form-encoded processing
And as a result, no need for `import urlparse`
2022-08-18 16:14:25 +01:00
Athanasius 1f991e7c44 Gateway: Remove un-necessary import urlparse 2022-08-18 16:13:59 +01:00
Athanasius 1384241f13 Fix "are you running this correctly?" check to utilise pathlib
As it was this only worked on systems with `/`-delimited path components.

So use `pathlib.Path.as_posix()` to ensure it *is* in that format.
2022-08-18 15:20:22 +01:00
Athanasius 98603273cc requirements.txt: Add strict-rfc3339==0.7 2022-08-18 15:20:21 +01:00
Athanasius 2310d9cb46 Relay: Add 'are you running this correctly?' check 2022-08-18 15:20:21 +01:00
Athanasius 86cfed0ec5 Monitor: Add 'are you running this correctly?' check 2022-08-18 15:20:21 +01:00
Athanasius 3f024f965c Bouncer: Add 'are you running this correctly?' check 2022-08-18 15:20:21 +01:00
Athanasius 6a4bad49c8 Gateway: Add 'are we running this correctly?' check
Also some misc import order fixups
2022-08-18 15:20:20 +01:00
Athanasius fb53653430 pre-commit: Specify more flake8 dependencies 2022-08-18 15:20:20 +01:00
Athanasius 1afa0b9c44 pre-commit: Specify flake8-isort dependency 2022-08-18 15:20:20 +01:00
Athanasius b9e97a15df pre-commit: Change flake8 to repo-based 2022-08-18 15:20:20 +01:00
Athanasius 84a19811aa pre-commit: Additional mypy dependencies & update versions 2022-08-18 15:20:20 +01:00
Athanasius cba052de38 requirements: Sanitised with 'pip-chill' and bumped to 'eager' versions 2022-08-18 15:20:19 +01:00
Athanasius 1e4cb3b1ab Add helper scripts for running from source, and document them. 2022-08-18 15:20:19 +01:00
Athanasius b437df5af8 Move main scripts back into src/eddn/
So, yes, running any of these absolutely directly:

    cd src
    python eddn/Gateway.py

will fail.  But you *can*:

    cd src
    python -m eddn.Gateway

and they work perfectly fine.
2022-08-18 15:20:19 +01:00
Athanasius 242fdff00c docs: General update for switching to wheel for install 2022-08-18 15:20:19 +01:00
Athanasius a55d1c3616 docs/Running: Assume python3 for virtualenv 2022-08-18 15:20:19 +01:00
Athanasius 7ad46d409b Rename main scripts to have "EDDN_" prefix
This avoids having possibly clashing files in the environment
sites-packages/, i.e. 'Gateway.py' is a bit too generic.
2022-08-18 15:20:18 +01:00
Athanasius 0f04fd61a5 All change yet again: We're now building and installing a wheel, not an egg
# Conflicts:
#	src/eddn/schemas/fssbodysignals-README.md
#	src/eddn/schemas/fssbodysignals-v1.0.json
#	src/eddn/schemas/fsssignaldiscovered-README.md
#	src/eddn/schemas/fsssignaldiscovered-v1.0.json
2022-08-18 15:20:18 +01:00
Athanasius ef2c184710 setup.py: Remove comment about pinning module version, as we're not
And we're on python3 now anyway.
2022-08-18 15:20:18 +01:00
Athanasius c2dc30c8a2 Finally got the correct setup.py configuration for source of egg running
* py_modules parameter to setup() isn't documented in the setuptools
  docs, but is in a general 'python packaging' one.
* So now the main scripts are NOT within the `eddn` package..
* But all other code is...
* But the schema files don't need to be.

# Conflicts:
#	src/schemas/fssbodysignals-README.md
#	src/schemas/fssbodysignals-v1.0.json
#	src/schemas/fsssignaldiscovered-README.md
#	src/schemas/fsssignaldiscovered-v1.0.json
2022-08-18 15:20:18 +01:00
Athanasius 47ae96ddc1 More file re-arrangement, concentrating on running from source
With `import eddn.core.Validator` and the like it was actually picking
up the egg versions of the files, not local ones.

Currently this will run as per:

cd src/eddn
python Gateway.py --config ~/.local/share/eddn/dev/config.json

# Conflicts:
#	src/eddn/schemas/fssbodysignals-README.md
#	src/eddn/schemas/fssbodysignals-v1.0.json
#	src/eddn/schemas/fsssignaldiscovered-README.md
#	src/eddn/schemas/fsssignaldiscovered-v1.0.json
2022-08-18 15:20:18 +01:00
Athanasius d36e45873d Initial re-arrangement of files
* Move the service scripts out of src/eddn/ into src/
* Move the schemas/ to src/schemas/
* Some updates to setup.py, i.e. source of files, but the setuptools
  part isn't yet work.

# Conflicts:
#	src/schemas/fssbodysignals-README.md
#	src/schemas/fssbodysignals-v1.0.json
#	src/schemas/fsssignaldiscovered-README.md
#	src/schemas/fsssignaldiscovered-v1.0.json
2022-08-18 15:20:17 +01:00
Athanasius ede9abb95e contrib: apache-eddn.conf: Attempt to correct for new "no direct access"
As we've changed things to default to *not* doing TLS on the actual
services, and thus relying on reverse proxying of all the relevant
end points, this needed updating.

Not yet tested, that will come when I put this on dev.eddn.edcd.io.
2022-08-18 15:20:17 +01:00
Athanasius 8ca813c6ab monitor: Make example eddn-config.js fit the usual live setup 2022-08-18 15:20:17 +01:00
Athanasius 0473e18633 monitor/schemas.html: Actually, those bits are schema URLs, so revert change
It's only the monitor endpoint URL that needed to be configurable.
Anyone using this wholesale for schemas not under eddn.edcd.io gets to
make the necessary hard-coded changes.
2022-08-18 15:20:17 +01:00
Athanasius f96e7fc293 monitor/schemas.html: Now supports eddn-config.js 2022-08-18 15:20:16 +01:00
Athanasius b81931fd3c setup.py: Backup/restore monitor eddn-config.js if present 2022-08-18 15:20:16 +01:00
Athanasius db698d70cf docs/Running: Update for monitor eddn-config.json
Although now I realise the schemas.html is going to need some work.
2022-08-18 15:20:16 +01:00
Athanasius e01b81b3b9 monitor: Move eddn-config.js to only an example, and make it such
We won't want the live config copied in/overwritten on each invocation
of 'python setup.py install'.
2022-08-18 15:20:16 +01:00
Athanasius 8f73060c74 monitor: Move configuration into separate file
This will allow for running more than one monitor without editing actual
source.
2022-08-18 15:20:16 +01:00
Athanasius 1ae349a4b7 Make per-component logging more obvious 2022-08-18 15:20:15 +01:00
Athanasius 4d705e4f37 Settings: Default Relay and Monitor to localhost listen. 2022-08-18 15:20:15 +01:00
Athanasius 95b56ee3b0 Relay: Allow for running without TLS 2022-08-18 15:20:15 +01:00
Athanasius d79f4a5aa1 Monitor: Change to be able to run TLS-less
NB: Not yet changed the actual monitor web page files.
2022-08-18 15:20:15 +01:00
Athanasius 128dab965c TLS: Optionally use TLS if you set non-empty CERT_FILE and KEY_FILE
Whilst we do want to go TLS-less in the actual EDDN code, for ease of
setting up automated end to end functional testing, leave the
possibility of running with TLS termination as well.
2022-08-18 15:20:15 +01:00
Athanasius 83e2b548ad Gateway: Use a dict for kwargs to app.run()
This is so we can now adjust if we're putting the TLS cert/key files in.
2022-08-18 15:20:14 +01:00
Athanasius d5dbc3262f Gateway: Put remote_addr in [], as it could be multiple, comma-separated
# Conflicts:
#	src/eddn/Gateway.py

# Conflicts:
#	src/eddn/Gateway.py
2022-08-18 15:20:12 +01:00