Commit Graph
202 Commits
Author SHA1 Message Date
Athanasius 613ef6deab Gateway: Properly report zlib.decompress() errors to uploaders
This also adds some debug/error logging to the code path.
2022-01-09 15:31:39 +00:00
Athanasius 1134a6c9b4 Gateway: Only log first 512 characters of invalid JSON
Don't want to spam the logs with up to 1MiB per bad message.
2022-01-06 17:39:01 +00:00
Athanasius 9f219da6a6 Gateway: Expand on logging for interpretation of errors
This includes logging the *full* (uncompressed) message if it fails to
parse as JSON.
2022-01-06 17:36:39 +00:00
Athanasius 1371f71217 Gateway: Set up proper logger formatting
For some reason the milliseconds portion of the %S timestamp is using a
comma for decimals separator, despite 'locale' saying we're set to (US)
English.  /tableflip
2022-01-06 14:06:59 +00:00
Athanasius 377bdd3833 Gateway: Add logging per Accepted request
Actually some logging was already there, just the logger had never been
set up properly, but then I decided to make the format of this message
more useful.
2022-01-06 13:54:33 +00:00
Athanasius 0e80c76cb5 Gateway: Set bottle request limit to 1MiB 2022-01-06 13:07:53 +00:00
Athanasius 1b290ff53d Validator: Specify *which* schema failed to load 2021-10-14 14:00:14 +00:00
Athanasius bfdba0699e Settings.py: Add new schemas to GATEWAY_JSON_SCHEMAS 2021-09-23 18:34:11 +00:00
Athanasius ad56800822 Bouncer: Use Settings.py, and thus override files, rather than hard-coded 2021-07-10 10:30:05 +00:00
Athanasius 579e185b86 Bouncer: Minor docs edit & remove un-needed Validator import 2021-07-08 20:37:41 +00:00
Athanasius 555c9dd813 Bouncer: Accept messages as if a Gateway, but then forward them to another
This is to be used during migration, running on the old host so as to
forward all messages to the new Gateway.

The destination is hard-coded in `LIVE_GATEWAY_URL`.
2021-07-08 12:16:50 +00:00
Athanasius 1df2fb749b Relay: Add missing from bottle import ..., request, ... 2021-07-02 17:07:46 +00:00
Athanasius 1e9edf94f7 Relay: app.-ise and CORS-ise
* Use the "make an explicit Bottle() and use it" change.
* Use app.route with OPTIONS in method= list.
* Remove extraneous setting of Access-Control-Allow-Origin header.
2021-07-02 17:02:53 +00:00
Athanasius 9359b78301 Gateway: Correct methodS to singular & remove extraneous header
* methods != method
* There shouldn't be any need to manually set
  Access-Control-Allow-Origin headers now.
2021-07-02 17:02:07 +00:00
Athanasius a8b3fc33e1 Monitor: Forgot the app.install(EnableCors()) 2021-07-02 16:57:06 +00:00
Athanasius 1feb7c2dee Monitor: Converted to app. including CORS handler 2021-07-02 16:54:37 +00:00
Athanasius 2eb8a9fb88 Gateway: Implement a CORS plugin for all routes
* Switches all of the routes to use .route(..., method=[...]) so as to
  include OPTIONS as well as either POST or GET.
2021-07-02 16:32:19 +00:00
AnthorNet e4bc72abe9 Remove GA 2020-07-06 14:24:09 +02:00
Athanasius 6d9942ece0 Switch bottle to using an 'app' instance.
This fixes the problem I was having (on two separate machines, Debian
stretch and Debian buster) with the Gateway not actually sending
messages out port 8500 to the Relay and Monitor.
  Something about the '@thing' syntax, or using bare 'run()' must be
interfering with zmq.green.  The latter ends up thinking there are no
active/matching sockets to send to[0], despite the sockets definitely
being there (complete with TCP 3-way handshake visible on tcpdump
output).
  With the problem:

	* no network traffic was observed on port 8500.
	* A test sender.send(...) just before the bottle run() call
	  *did* send the message.  A similar test at the start of the
	  @post('/upload/') function did not succeed.

[0] - I ended up putting debug prints in both python-zmq and the zeromq
'libzmq' libraries, building the Debian packages and installing those
versions.  I also ended up using 'gdb' on the process.  The end result
of this was to find that the _matching variable (a count of matching
sockets I think) was empty deep in libzmq, when it should be counting
sockets to send to.
  This was specifically in zeromq3-4.3.1/src/dist.cpp, in the
void zmq::dist_t::distribute (msg_t *msg_) function.  The immediate:

	    if (_matching == 0) {

test was true.  I didn't manage to track down which bit of libzmq code
should have been setting _matching before I 'recursed back up' the call
chain to investigate other things.
2019-10-05 11:23:41 +01:00
AnthorNet b21bdf76e5 Merge pull request #90 from Marginal/tidy
Remove redundant editing of journal faction info and clean up duplicate detection code
2019-04-29 15:32:55 +02:00
AnthorNet 9484f4646a Fix for UTF8 uploader 2019-04-29 15:31:10 +02:00
AnthorNet 1150a72b47 Remove uploader from monitor 2019-01-12 10:22:39 +01:00
Jonathan Harris be7fcc2c05 Remove redundant cleanup of journal faction info
Clean up duplicate detection code
2019-01-08 18:58:41 +00:00
AnthorNet 7b45a4ce78 Also comment in monitor 2018-12-14 12:08:05 +01:00
AnthorNet 42127cb43a Try High Water Mark on Relay 2018-12-14 11:58:38 +01:00
AnthorNet e4c3e61b0f Also remove DistanceFromArrivalLS 2018-12-13 13:47:51 +01:00
AnthorNet b5eb8bc2ce Avoid duplicate scan messages after SAAScanComplete 2018-12-13 13:40:25 +01:00
AnthorNet a54b0bd04e Remove duplicate timestamp to avoid close messages to pass 2018-12-13 09:01:01 +01:00
AnthorNet a733ff22ed Fix MyReputation deletion 2018-11-02 12:30:07 +01:00
AnthorNet a8bf0b2f0b Do not send faction reputation to listeners 2018-10-30 20:11:57 +01:00
AnthorNet f183391402 Update SQLLite query to MySQL 2018-10-23 11:44:11 +02:00
AnthorNet 5b74875630 Migrate SQLLite to MySQL 2018-10-23 09:58:21 +02:00
Ian Norton d74505fe35 Ensure that the uploaderID value is always unique for each uploader but is also unpredictable.
Also ensure that the unique uploaderID for each uploader changes every 12 hours
2018-03-07 22:52:55 +00:00
AnthorNet d23a7dc722 Always hash uploaderID with private information 2018-01-30 07:31:23 +01:00
AnthorNet a221ae13c3 Remove uploaderID from public relays to avoid real time tracking of user without consent 2018-01-27 09:37:26 +01:00
AnthorNet 08eb0aa28f Only BIND Gateway ton localhost to avoid unintended listening of unfiltered messages 2018-01-27 09:14:43 +01:00
Jonathan Harris e5c4233939 Don't modify message body
Fixes 6d78549
2018-01-07 02:23:01 +00:00
AnthorNet 63cac32704 Merge pull request #70 from Marginal/master
Make duplicate test cheaper
2018-01-06 12:41:22 +01:00
AnthorNet 26f0d64745 Small fixes 2018-01-06 12:40:56 +01:00
Jonathan Harris 6d78549655 Make duplicate test cheaper 2018-01-05 20:31:44 +00:00
AnthorNet ad1d133e66 Fix (Ignore): You must not use 8-bit bytestrings 2018-01-05 16:41:19 +01:00
AnthorNet 32ed86da91 Fix duplicate message not copied 2018-01-05 16:35:57 +01:00
AnthorNet dd65ba50d1 Temp fix 2018-01-05 15:10:53 +01:00
AnthorNet 6d3b593fd1 Fix json is not defined 2018-01-05 15:09:27 +01:00
AnthorNet 313120aa94 Implement tracking stats 2018-01-05 15:06:21 +01:00
AnthorNet 4fad830e59 Avoid KeyError 2017-12-28 11:23:37 +01:00
AnthorNet e5123b99b5 Update DuplicateMessages.py 2017-12-28 11:06:28 +01:00
AnthorNet 6d670a27ee Prevent docked dup eon distance from start 2017-12-28 11:00:59 +01:00
AnthorNet 1443e45272 No need to divide again, INT are enough 2017-12-28 10:28:44 +01:00
AnthorNet 2770d37f16 Update DuplicateMessages.py 2017-12-28 10:24:38 +01:00