714 Commits

Author SHA1 Message Date
Athanasius
fe214583aa Relay: Support --loglevel <level> fully
There's no real logging in Relay yet, but this makes it support
switching level anyway.
2022-01-18 15:16:32 +00:00
Athanasius
86b283f920 Monitor.py: Minimal changes to allow --loglevel <level>
The CL arg won't actually *do* anything yet, because this script doesn't
actually have a logger set up at all.  But we need to fake this else
eddn.conf.Settings.loadConfig() won't be called correctly or work.
2022-01-18 15:09:56 +00:00
Athanasius
fb3c0348ad Gateway: Support --loglevel <level> CL arg
NB: This required changes to src/eddn/conf/Settings.py as well, which
means other scripts, e.g. Monitor.py, will also need changing.
2022-01-18 15:02:38 +00:00
Athanasius
872c65c280 setup.py: Do *not* install eddn-logs-archive script
This should be done manually, so as to be using the correct version from
the correct branch on the live service.
2022-01-18 14:16:45 +00:00
Athanasius
79b78ac831 scripts/eddn-report: Quieten some EDMC errors
They're noted in <https://github.com/EDCD/EDMarketConnector/issues/1403>
2022-01-18 11:21:21 +00:00
Athanasius
3ffff54c4b contrib/eddn-logs-archive: Run gateway error report script on gateway.log
* Renamed some variables to be truer to their content and use.
* Run eddn-report-log-errors if rotating a gateway.log file.
2022-01-17 09:22:37 +00:00
Athanasius
a1e330be2c scripts/eddn-report: EVA is abandoned, so ignore 2022-01-17 09:18:40 +00:00
Athanasius
cc5ae32227 scripts/eddn-report: Correct EliteLogAgent comment 2022-01-12 15:43:59 +00:00
Athanasius
e93c04bb21 scripts/eddn-report: EliteLogAgent latest is 2.0.0.660
This allows for ignoring the ancient 0.9.2.412 that showed some errors
yesterday.
2022-01-12 09:40:47 +00:00
Athanasius
00071ba7e9 scripts/testing: Remove per-scenario scripts
The required functionality is now all in `test-sender.py`.
2022-01-11 16:43:30 +00:00
Athanasius
ad8fc57df3 scripts/test-sender: Support --gzip=bad 2022-01-11 16:41:25 +00:00
Athanasius
7537a300c0 scripts/test-sender: Support for valid gzip compression 2022-01-11 16:31:05 +00:00
Athanasius
fb83aae240 scripts/test-sender: Support for requesting good or bad form encoding 2022-01-11 16:27:20 +00:00
Athanasius
ecd3afbbbc scripts/test-sender.py: Allow 'beta' and 'dev' --url aliases
Also explicitly states the URL and input file it uses.
2022-01-11 16:12:21 +00:00
Athanasius
02b2ea6e18 scripts/test-sender.py: Allow for changing target URL 2022-01-11 16:08:59 +00:00
Athanasius
a4eb7548d9
Merge pull request #166 from EDCD/fix/165/bad-formencoded-detection
Gateway: Revert non-gzip form encoded check
2022-01-11 15:39:01 +00:00
Athanasius
6799a7fae9 scripts/testing: Target dev:4432, not beta:4431
In the long-term these all need to support a CL arg to select the
target.
2022-01-11 15:36:53 +00:00
Athanasius
a1d347aec5 scripts/testing: Add general "uses gzip" script 2022-01-11 15:35:24 +00:00
Athanasius
abcf472197 Gateway: Revert non-gzip form encoded check
Fixes the regression identified in #165

However the gzip code path can still erroneously think a decompressed
request body is form encoded when it is not.  This happens when any text
in the decompressed body matches the regex:

        .+=[^\&;]*

i.e. some text, followed by `=`, and then some more text, possibly
empty, followed by an ampersand `&`, or semi-colon `;`, or the end of the
string.

`&` and `;` are used to separate key=value pairs in form encoding, the
`=` separates a key from its value.
2022-01-11 15:06:02 +00:00
Athanasius
6add1d5c6b Merge branch 'master' of github.com:EDCD/EDDN 2022-01-11 15:01:20 +00:00
Athanasius
759b14c844 scripts/testing: Add "valid JSON looks like form-encoded" file
Ref: https://github.com/EDCD/EDDN/issues/165
2022-01-11 14:40:16 +00:00
Athanasius
2d60092d0e
README: Emphasise developers must comply with documentation 2022-01-11 13:47:41 +00:00
Athanasius
1caf5f0600
README: 'Documentation' section removed as un-necessary 2022-01-11 13:44:42 +00:00
Athanasius
01ce718ab8
Merge branch 'master' of github.com:EDSM-NET/EDDN 2022-01-11 13:42:48 +00:00
Athanasius
40262322ff
README: Expansion and tweaks for replacing the Wiki page
* A few uploading PC softwares are listed to lower the barrier for a new
  player finding such.  This is *never* going to be an exhaustive list.
* Added section listing console options for contributing data.  Again,
  not intended to be exhaustive.
* Layout/sections tweaked.  Hopefully the use of double horizontal rules
  emphasises which section text is a part of.
2022-01-11 13:40:42 +00:00
Athanasius
a39f24c55d
README: Add 'Contact' section at the bottom 2022-01-11 13:13:06 +00:00
Athanasius
deb7383e32 docs/schemas: Additions and tweaks for uploaders
* Document the "you tried to use plain HTTP" response from Reverse
Proxy in the appropriate section.

* Switch URL for Gateway.py to `live` branch.

* 'disallowed' Schema Validation error cites **value**, not key *name*.

* Call out the possible need to explicitly subscribe to the empty topic
in order to receive messages... and that currently server-side topic
filtering isn't possible.
2022-01-11 12:48:40 +00:00
Athanasius
1943003b06
README: Complete re-write
* Expand the initial blurb into a full 'About' section.
* Making 'Using EDDN' an actual section, to facilitate migrating away
  from the wiki page.
* Guide players towards some useful information (guide and sites).
* Ensure developers know to check the *live* documentation.
* Explicitly link to the current wiki page and EDDN status.
* Add 'Hosting of the live service' section, and only list the current
  situation (Vivio hosting is 4-5 years ago now, and Anthor's was never
  called out).
2022-01-11 12:24:33 +00:00
Athanasius
084ea35286 setup.py: Change "live branch for live" check to use setup_env.py
The whole build/install process is dependent on the EDDN_ENV value
anyway, so we might as well use it.

This way if a future setup doesn't have a specific string in CWD this
check doesn't break.

Yes, this whole check means you MUST be building from a git checkout,
not the files sourced by some other means.
2022-01-10 16:23:14 +00:00
Athanasius
35c4f89c68 setup.py: Detect if not using live branch in live directory 2022-01-10 16:17:39 +00:00
Athanasius
495e27861e scripts: Make it obvious that test-bad-gzip.py doesn't work 2022-01-10 15:53:44 +00:00
Athanasius
a8098f45b9 Move eddn-report-log-errors into scripts/ 2022-01-10 09:42:17 +00:00
Athanasius
aa7fabf134 Merge branch 'master' into develop 2022-01-10 09:40:14 +00:00
Athanasius
66d39a1bba Merge branch 'master' into develop 2022-01-10 09:38:53 +00:00
Athanasius
f31974d6ff
Merge pull request #162 from EDCD/enhancement/161/gateway-improve-misc-error-reports
Gateway: Properly report all errors to uploaders
2022-01-09 16:41:15 +00:00
Athanasius
ddb9480ee6
Gateway/testscripts: Add a small README 2022-01-09 16:27:08 +00:00
Athanasius
2a8eb8d021
Gateway: Add test scripts and supporting files
These are what I was using on my home server to test the prior code
changes to how the Gateway code reports errors.

Ultimately these should become a part of proper tests, but for now
they're at least in the repository for anyone to utilise.
2022-01-09 16:21:23 +00:00
Athanasius
75c117fb12
docs/schemas: form-encoded messages are deprecated and not documented
At some point I might add logging to see if any sender even *is*
utilising this, and if not actually remove the possibility entirely.
2022-01-09 16:14:00 +00:00
Athanasius
11d3501c59
Gateway/errors: More documentation tweaking 2022-01-09 16:13:01 +00:00
Athanasius
4ad1519d1d
Gateway/errors: Improve general outline of documentation section 2022-01-09 16:10:01 +00:00
Athanasius
ccde820ba7
Gateway/errors: 'Schema Validation' properly tagged and documented 2022-01-09 16:02:28 +00:00
Athanasius
e111fb8415
Gateway/errors: Outdate Schema: Tweak message and document 2022-01-09 15:48:27 +00:00
Athanasius
e931bfff96
Gateway/errors: MalformedUploadError isn't raised upon JSON issue 2022-01-09 15:45:56 +00:00
Athanasius
b9b59329d5
Gateway/errors: Document 'Malformed Upload' error 2022-01-09 15:43:51 +00:00
Athanasius
4da60215f0
Gateway/errors: Tag FAIL if a zlib.error 2022-01-09 15:43:12 +00:00
Athanasius
8edae919e2
Gateway/errors: Call out if FAIL is specifically on JSON parsing 2022-01-09 15:38:29 +00:00
Athanasius
35cd3c3294
Merge branch 'enhancement/161/gateway-improve-misc-error-reports' of github.com:EDSM-NET/EDDN into enhancement/161/gateway-improve-misc-error-reports 2022-01-09 15:33:24 +00:00
Athanasius
81a70572c9 Gateway: Properly report 'not compressed, badly form-encoded' to uploaders
* This code worked if the request was *properly* form-encoded, with a
'data' key whose value was a valid message.

* It failed to detect where the request was form-encoded, with without a
'data' key.  It would just assume 'not form-encoded' in that case, then
fail later on JSON parsing.

Thus, re-use the `urlparse.parse_qs()` check for form-encoded format.
This passes:

  1. Properly, `data` key, form-encoded with valid value is fully JSON
  parsed, schema checked and accepted.
  2. *NOT* compressed *or* form-encoded valid message is properly parsed
  and accepted.
  2. Uncompressed, form-encoded, but no `data` key correctly returns the
  same error status and body as the compressed+form-encoded+no data key
  path.
2022-01-09 15:31:39 +00:00
Athanasius
faa2e25d62 Gateway: Properly report 'badly form encoded' to uploaders
This currently only applies if the data was also, validly, compressed.
2022-01-09 15:31:39 +00:00
Athanasius
613ef6deab Gateway: Properly report zlib.decompress() errors to uploaders
This also adds some debug/error logging to the code path.
2022-01-09 15:31:39 +00:00