202 Commits

Author SHA1 Message Date
Athanasius
613ef6deab Gateway: Properly report zlib.decompress() errors to uploaders
This also adds some debug/error logging to the code path.
2022-01-09 15:31:39 +00:00
Athanasius
1134a6c9b4 Gateway: Only log first 512 characters of invalid JSON
Don't want to spam the logs with up to 1MiB per bad message.
2022-01-06 17:39:01 +00:00
Athanasius
9f219da6a6 Gateway: Expand on logging for interpretation of errors
This includes logging the *full* (uncompressed) message if it fails to
parse as JSON.
2022-01-06 17:36:39 +00:00
Athanasius
1371f71217 Gateway: Set up proper logger formatting
For some reason the milliseconds portion of the %S timestamp is using a
comma for decimals separator, despite 'locale' saying we're set to (US)
English.  /tableflip
2022-01-06 14:06:59 +00:00
Athanasius
377bdd3833 Gateway: Add logging per Accepted request
Actually some logging was already there, just the logger had never been
set up properly, but then I decided to make the format of this message
more useful.
2022-01-06 13:54:33 +00:00
Athanasius
0e80c76cb5 Gateway: Set bottle request limit to 1MiB 2022-01-06 13:07:53 +00:00
Athanasius
1b290ff53d Validator: Specify *which* schema failed to load 2021-10-14 14:00:14 +00:00
Athanasius
bfdba0699e Settings.py: Add new schemas to GATEWAY_JSON_SCHEMAS 2021-09-23 18:34:11 +00:00
Athanasius
ad56800822 Bouncer: Use Settings.py, and thus override files, rather than hard-coded 2021-07-10 10:30:05 +00:00
Athanasius
579e185b86 Bouncer: Minor docs edit & remove un-needed Validator import 2021-07-08 20:37:41 +00:00
Athanasius
555c9dd813 Bouncer: Accept messages as if a Gateway, but then forward them to another
This is to be used during migration, running on the old host so as to
forward all messages to the new Gateway.

The destination is hard-coded in `LIVE_GATEWAY_URL`.
2021-07-08 12:16:50 +00:00
Athanasius
1df2fb749b Relay: Add missing from bottle import ..., request, ... 2021-07-02 17:07:46 +00:00
Athanasius
1e9edf94f7 Relay: app.-ise and CORS-ise
* Use the "make an explicit Bottle() and use it" change.
* Use app.route with OPTIONS in method= list.
* Remove extraneous setting of Access-Control-Allow-Origin header.
2021-07-02 17:02:53 +00:00
Athanasius
9359b78301 Gateway: Correct methodS to singular & remove extraneous header
* methods != method
* There shouldn't be any need to manually set
  Access-Control-Allow-Origin headers now.
2021-07-02 17:02:07 +00:00
Athanasius
a8b3fc33e1 Monitor: Forgot the app.install(EnableCors()) 2021-07-02 16:57:06 +00:00
Athanasius
1feb7c2dee Monitor: Converted to app. including CORS handler 2021-07-02 16:54:37 +00:00
Athanasius
2eb8a9fb88 Gateway: Implement a CORS plugin for all routes
* Switches all of the routes to use .route(..., method=[...]) so as to
  include OPTIONS as well as either POST or GET.
2021-07-02 16:32:19 +00:00
AnthorNet
e4bc72abe9 Remove GA 2020-07-06 14:24:09 +02:00
Athanasius
6d9942ece0 Switch bottle to using an 'app' instance.
This fixes the problem I was having (on two separate machines, Debian
stretch and Debian buster) with the Gateway not actually sending
messages out port 8500 to the Relay and Monitor.
  Something about the '@thing' syntax, or using bare 'run()' must be
interfering with zmq.green.  The latter ends up thinking there are no
active/matching sockets to send to[0], despite the sockets definitely
being there (complete with TCP 3-way handshake visible on tcpdump
output).
  With the problem:

	* no network traffic was observed on port 8500.
	* A test sender.send(...) just before the bottle run() call
	  *did* send the message.  A similar test at the start of the
	  @post('/upload/') function did not succeed.

[0] - I ended up putting debug prints in both python-zmq and the zeromq
'libzmq' libraries, building the Debian packages and installing those
versions.  I also ended up using 'gdb' on the process.  The end result
of this was to find that the _matching variable (a count of matching
sockets I think) was empty deep in libzmq, when it should be counting
sockets to send to.
  This was specifically in zeromq3-4.3.1/src/dist.cpp, in the
void zmq::dist_t::distribute (msg_t *msg_) function.  The immediate:

	    if (_matching == 0) {

test was true.  I didn't manage to track down which bit of libzmq code
should have been setting _matching before I 'recursed back up' the call
chain to investigate other things.
2019-10-05 11:23:41 +01:00
AnthorNet
b21bdf76e5
Merge pull request #90 from Marginal/tidy
Remove redundant editing of journal faction info and clean up duplicate detection code
2019-04-29 15:32:55 +02:00
AnthorNet
9484f4646a
Fix for UTF8 uploader 2019-04-29 15:31:10 +02:00
AnthorNet
1150a72b47 Remove uploader from monitor 2019-01-12 10:22:39 +01:00
Jonathan Harris
be7fcc2c05 Remove redundant cleanup of journal faction info
Clean up duplicate detection code
2019-01-08 18:58:41 +00:00
AnthorNet
7b45a4ce78 Also comment in monitor 2018-12-14 12:08:05 +01:00
AnthorNet
42127cb43a Try High Water Mark on Relay 2018-12-14 11:58:38 +01:00
AnthorNet
e4c3e61b0f Also remove DistanceFromArrivalLS 2018-12-13 13:47:51 +01:00
AnthorNet
b5eb8bc2ce Avoid duplicate scan messages after SAAScanComplete 2018-12-13 13:40:25 +01:00
AnthorNet
a54b0bd04e Remove duplicate timestamp to avoid close messages to pass 2018-12-13 09:01:01 +01:00
AnthorNet
a733ff22ed Fix MyReputation deletion 2018-11-02 12:30:07 +01:00
AnthorNet
a8bf0b2f0b Do not send faction reputation to listeners 2018-10-30 20:11:57 +01:00
AnthorNet
f183391402 Update SQLLite query to MySQL 2018-10-23 11:44:11 +02:00
AnthorNet
5b74875630 Migrate SQLLite to MySQL 2018-10-23 09:58:21 +02:00
Ian Norton
d74505fe35 Ensure that the uploaderID value is always unique for each uploader but is also unpredictable.
Also ensure that the unique uploaderID for each uploader changes every 12 hours
2018-03-07 22:52:55 +00:00
AnthorNet
d23a7dc722 Always hash uploaderID with private information 2018-01-30 07:31:23 +01:00
AnthorNet
a221ae13c3 Remove uploaderID from public relays to avoid real time tracking of user without consent 2018-01-27 09:37:26 +01:00
AnthorNet
08eb0aa28f Only BIND Gateway ton localhost to avoid unintended listening of unfiltered messages 2018-01-27 09:14:43 +01:00
Jonathan Harris
e5c4233939 Don't modify message body
Fixes 6d78549
2018-01-07 02:23:01 +00:00
AnthorNet
63cac32704
Merge pull request #70 from Marginal/master
Make duplicate test cheaper
2018-01-06 12:41:22 +01:00
AnthorNet
26f0d64745 Small fixes 2018-01-06 12:40:56 +01:00
Jonathan Harris
6d78549655 Make duplicate test cheaper 2018-01-05 20:31:44 +00:00
AnthorNet
ad1d133e66 Fix (Ignore): You must not use 8-bit bytestrings 2018-01-05 16:41:19 +01:00
AnthorNet
32ed86da91 Fix duplicate message not copied 2018-01-05 16:35:57 +01:00
AnthorNet
dd65ba50d1 Temp fix 2018-01-05 15:10:53 +01:00
AnthorNet
6d3b593fd1 Fix json is not defined 2018-01-05 15:09:27 +01:00
AnthorNet
313120aa94 Implement tracking stats 2018-01-05 15:06:21 +01:00
AnthorNet
4fad830e59
Avoid KeyError 2017-12-28 11:23:37 +01:00
AnthorNet
e5123b99b5
Update DuplicateMessages.py 2017-12-28 11:06:28 +01:00
AnthorNet
6d670a27ee
Prevent docked dup eon distance from start 2017-12-28 11:00:59 +01:00
AnthorNet
1443e45272
No need to divide again, INT are enough 2017-12-28 10:28:44 +01:00
AnthorNet
2770d37f16
Update DuplicateMessages.py 2017-12-28 10:24:38 +01:00