1
0
mirror of https://github.com/EDCD/EDMarketConnector.git synced 2025-04-14 00:07:14 +03:00

4895 Commits

Author SHA1 Message Date
Athanasius
d902093f0b
Merge pull request #1682 from EDCD/fix/1431/eddn-replay-enhancements
Change EDDN 'replay' to using an sqlite3 DB & otherwise improve
2022-11-27 15:24:28 +00:00
Athanasius
642d1912d8
Merge pull request #1716 from EDCD/enhancement/1715/inara-live-galaxy-only
inara: Send only Live galaxy data after Update 14
2022-11-27 13:55:35 +00:00
Athanasius
b3eff6ab32
monitor: Catch any failure to coerce gameversion
Also, log either the error or the success.  That will aid us in diagnosing
any future issues.
2022-11-27 13:02:54 +00:00
Athanasius
8a58220a66
inara: Send only Live galaxy data after Update 14
* Adds `monitor.is_live_galaxy()` for general use.
* Assumes Update 14 starts after 2022-11-29T09:00:00+00:00.  That's the
  currently schedule day, and recently the servers have been down by the time.
  Likelihood of them coming back *up* quickly seems slim to none.
* If we couldn't parse the `gameversion` from Journal using
  `semantic_version.Version.coerce()` this will fail, and assume we're on
  the Legacy galaxy.
2022-11-27 12:47:19 +00:00
Athanasius
345938a81d
eddn: Set gamebuild="" for CAPI-sourced data
Whilst setting it to the same "CAPI-<endpoint>" string as `gameversion` in
these cases would probably be OK, that's not the intent of the EDDN
documentation, which has now been clarified.
2022-11-25 15:45:07 +00:00
Athanasius
4d29eb6059
appversion: Change to 5.6.0-alpha0 for 'new EDDN code' testing
* There's a Pre-Release 5.5.1-alpha0 out there, we want to look distinct
  from it.
2022-11-24 14:42:34 +00:00
Athanasius
073afc842f
eddn: New --trace-on plugins.eddn.send & other logging tweaks
* In case of apparent issues, have a `--trace-on` to better see what's (not)
  happening.

  All the old DBEUG logging, even if commented out, is now under this.
* Also added some INFO level logging for the legacy replay.jsonl conversion,
  as it should be one-time per user.
* Some additional DEBUG logging for closing down.
2022-11-24 14:09:57 +00:00
Athanasius
75af13efea
Merge pull request #1710 from der-eismann/change-eddn-wiki-url
Fix URL for EDDN GitHub Link
2022-11-24 13:32:57 +00:00
Philipp Trulson
c8e00304fb
Fix URL for EDDN GitHub Link 2022-11-24 14:09:43 +01:00
Athanasius
32229217b2
eddn: Bail from sending queued messages if one failed
NB: This is failed *not* due to the message being 'bad' in some manner. It
will mean the Gateway timed out, refused connection etc.
2022-11-23 16:00:28 +00:00
Athanasius
0eb33e011b
PLUGINS.md: Document new IsDocked state flag. 2022-11-23 15:57:25 +00:00
Athanasius
95fa9d577c
eddn: Use a shorter, 10 second, delay at startup for first queue check 2022-11-23 14:20:24 +00:00
Athanasius
3d9bb64330
eddn: Remove outdated comment about replaylog rework
And using tk `parent` *is* absolutely necessary.
2022-11-23 14:12:43 +00:00
Athanasius
b7769821be
eddn: Send immediately if docked
This has the "are we delaying sending?" check, but without the short-circuit
for "we are actually docked right now".
2022-11-23 13:56:01 +00:00
Athanasius
524c0425fb
eddn: EDDN.send_message(): Only set standard header if not already set.
Else we risk overwriting e.g. `CAPI-shipyard`.
2022-11-23 13:46:49 +00:00
Athanasius
fda91df04f
eddn: Working with tk after(), on timer or when docked
* An aborted attempt was made to use a thread worker, but:
  1. sqlite3 doesn't allow cross-thread use of the same sqlite3 connection.
  2. Having an on-going query on one cursor, e.g. gathering all the
    outstanding message `id`, whilst trying to DELETE a row hits a
    "database is locked" error.
* So, back to tk `after()`.  `send_message_by_id()` has been audited to ensure
  its boolean return is accurate.  So there shouldn't be any way in which to
  get hung up on a single message *other than if the EDDN Gateway is having
  issues, and thus it should be retried anyway*.  Any reason for a 'bad
  message' will cause `True` return and thus deletion of the message in
  *this* call to `queue_check_and_send()`.
* There is a new `reschedule` parameter to `queue_check_and_send()`.  If
  `True` then at the end it should re-schedule.

  There is a check in `journal_entry()` for the `Docked` event, and if this
  occurs it will schedule `queue_check_and_send()` with `reschedule` set to
  `False` so that we don't end up with multiple parallel schedulings.

  It's still possible for a docking to have coincided with a scheduled run
  and thus cause double-rate sending to EDDN, but we can live with that.

* The same scheduling mechanism is used, with a much smaller delay, to
  process more than one queued message per run.

  Hence the `have_rescheduled` bool *in* the function to indicate if a 'fast'
  reschedule has already been set.  This prevents the slow one *also* being
  set in this scenario.  The latter will be scheduled when the fast one
  found no more rows to process.
2022-11-23 13:29:47 +00:00
Athanasius
f2dbfacf70
eddn: Add header for CAPI outfitting and shipyard exports 2022-11-22 17:40:33 +00:00
Athanasius
f6e2504216
eddn: Fix capi_fcmaterials gameversion/build & add to CAPI commodity
* Erroneously used 'CAPI-commoodity' when it's 'CAPI-market' (name of the
  CAPI endpoint, not anything to do with EDDN schema names, and '-commodity'
  would also be wrong for that).
* Set `header` for (CAPI) `export_commodities()`.
2022-11-22 17:37:15 +00:00
Athanasius
b31c8c0536
eddn: A start on ensuring header (gameversion/build) are always set 2022-11-22 17:35:17 +00:00
Athanasius
4a0518da9f
eddn: Set header up properly for all journal messages
* New function `EDDN.add_header()`.
* It utilises new `this` members, set from `journal_entry()`.
2022-11-22 17:26:32 +00:00
Athanasius
06fa3629ea
eddn: OUT_EDDN_DELAY (not inverted) & further work on the sending
* The eddn parts of the OUT_EDDN_DO_NOT_DELAY -> OUT_EDDN_DELAY change.  This
  includes the 'sense' of it being inverted from what it was.
* EDDN.REPLAY_DELAY is now a float, as it's used with `time.sleep()`.  *This*
  is the 400ms value for inter-message cooldown.
* EDDN.REPLAY_PERIOD is still an int, used with tk `after()`.  This is how
  often we attempt the queue.
* EDDN.session is no longer a thing, move that part of EDDN.close() to
  EDDNSender.close().
* EDDN queue DB has `id`, not `message_id`.
* Now *looping* in the queue sender, not only sending the oldest message.
2022-11-22 16:49:02 +00:00
Athanasius
6070f82c6b
OUT_EDDN_DO_NOT_DELAY should be OUT_EDDN_DELAY
* The old name was OUT_SYS_DELAY.
* Yes, this is the inverse of what we want, which is "should we not delay
  messages", but this is the legacy.
2022-11-22 16:47:17 +00:00
Athanasius
59f046ee67
eddn: First cut of periodic retry of sending messages 2022-11-22 15:51:10 +00:00
Athanasius
be1ef32238
config: Minor ordering change to ensure OUT_STATION_ANY set correctly
* This depends on `OUT_EDDN_SEND_STATION_DATA` which is defined below where
  this originally was.
2022-11-22 15:30:30 +00:00
Athanasius
a2d2723f49
monitor: Add an 'IsDocked' flag to monitor.state.
* This is cleaner than starting to track it in `plugins/eddn.py` specifically.
* This is literally only about if we're piloting a ship that is docked, so
  not even trying to resolve "on-foot, in a station, 'Location' said not
  docked though".
2022-11-22 15:29:05 +00:00
Athanasius
876c34ecfd
eddn: Remove two 'noqa: CCR001' that are no longer applicable 2022-11-22 14:09:08 +00:00
Athanasius
871f50288e
prefs.py: Fix overly long line (config constants renames) 2022-11-22 14:02:22 +00:00
Athanasius
06edcf3ea9
EDDN: Remove EDDN.send() in favour of renamed .export_journal_entry()
* Now that we're not trying to do "did we just/are we know docked?" in this
  code it turns out that both CAPI and Journal messages can use the same
  function for this.
* And as it's no longer journal-specific `EDDN.export_journal_entry()` has
  been renamed to `EDDN.send_message()`.

This whole branch now needs to actually implement sending queued messages
when docked, and periodically in the case of initial failures.
2022-11-22 14:02:21 +00:00
Athanasius
fe24cf7e95
EDDN: Remove legacy sendreplay()
* `EDDN.sendreplay()` is no longer used.
* In `prefsvarschanged()` there was a reference to `eddn.replayfile`, so as
  to grey out the "Delay sending..." option if the file wasn't available.
  So that's moot and also removed, but also...
* Comment the purpose of that line in `prefsvarchanged()` because it's not
  immediately obvious.
2022-11-22 14:02:20 +00:00
Athanasius
db7bb735d2
EDDN: Put "should we send (now?)" checks into EDDN.export_journal_entry()
In some cases the check might already have been done, but if not then this
is the last easy place to perform it.

NB: Unlike the old code this does *not* attempt to check "are we docked
  *now* ?" for triggering sending of previously queue messages.  That's
  going to need a thread worker.
2022-11-22 14:02:19 +00:00
Athanasius
03b36cbe39
EDDN: Improve export_journal_fcmaterials "don't send" comment 2022-11-22 14:02:18 +00:00
Athanasius
5eb4296ec6
EDDN: First steps to ensure new code respects user configuration
It's easier to check "should we send this message at all?" earlier. Currently
all of the following ('station data') do so:

* CAPI commodity, outfitting (also fcmaterials) and shipyard.
* Journal commodity, fcmaterials, outfitting,  and shipyard.
2022-11-22 14:02:17 +00:00
Athanasius
7d0ae88757
EDDN: EDDNSender.send_message() doesn't care about do/don't send options
By this point other code will have made that decision.
2022-11-22 14:02:16 +00:00
Athanasius
0d35f8874a
config/EDDN: Rename OUT_MKT_EDDN to OUT_EDDN_SEND_STATION_DATA
This flag controls whether commodity, outfitting or shipyard schema messages
are sent.  Thus 'MKT' ('market') is misleading.  Rename it so the intent when
used is clear.
2022-11-22 14:02:15 +00:00
Athanasius
9f02f18408
config/EDDN: Rename OUT_SYS_EDDN to OUT_EDDN_SEND_NON_STATION
* This was perhaps originally meant for what the UI option says, i.e. "send
  system and scan data", but is actually being used for anything that is
  **NOT** 'station data' (even though *that* option has 'MKT' it includes
  outfitting and shipyard as well).

  So, just name this more sanely such that code using it is more obvious as
  to the actual intent.
2022-11-22 14:02:14 +00:00
Athanasius
3a57b53bbd
config/EDDN: Rename OUT_SYS_DELAY to OUT_EDDN_DO_NOT_DELAY
The sense of this `output` flag has been inverted (always?) for a long time.

1. I have the option "Delay sending until docked" showing as *off* in the UI.
2. My config.output value is `100000000001`.
3. The value of this flag is `4096`, which means 12th bit (starting from 1, not
  zero).
4. So I have the bit set, but the option visibly off.

So, rename this both to be more pertinent to its use *and* to be correct as to
what `True` for it means.
2022-11-22 14:02:13 +00:00
Athanasius
598e54eaa4
EDDNSender: Now properly sends messages to Gateway
Including removing from the queue if it succeeded, or didn't and should be
dropped.
2022-11-22 14:02:12 +00:00
Athanasius
f66a98464e
EDDNSender: Closer to actually sending messages now 2022-11-22 14:02:11 +00:00
Athanasius
2b957d140c
EDDNSender: convert_legacy_file() belongs with "open the database" 2022-11-22 14:02:10 +00:00
Athanasius
0e20f4bc00
EDDNSender: Remove legacy file after migration 2022-11-22 14:02:09 +00:00
Athanasius
f3017d40ec
EDDNSender: Fill out type hints 2022-11-22 14:02:08 +00:00
Athanasius
51fb90b999
EDDN: Change EDDN.close() to call into EDDNSender.close() 2022-11-22 14:02:07 +00:00
Athanasius
c1793ad839
EDDN: Remove EDDN->flush() 2022-11-22 14:02:06 +00:00
Athanasius
09f646a249
EDDNSender: Add delete_message() method
This was tested by temporary code in `convert_legacy_file()` to delete
the last added row once all done.
2022-11-22 14:02:06 +00:00
Athanasius
86ff787aed
EDDNSender: Convert all of a legacy file
* I had a `break` in there to only convert the first message.
* Also collapsed the assignment to `cmdr, msg` to not go via `j`.
2022-11-22 14:02:05 +00:00
Athanasius
089c33002c
EDDNSender->add_message() returns ID of INSERTed row 2022-11-22 14:02:04 +00:00
Athanasius
80129361fe
EDDN: Refactor queue db open/creation to own function 2022-11-22 14:02:03 +00:00
Athanasius
9a660b3b49
EDDN: New class as EDDNSender now, and version the queue database file
* It makes more sense for this new class to be concerned with all the 'send it'
  functionality, not just 'replay', so rename it.
* Athough we're trying to get the schema right *first* time, let's plan ahead
  and version the filename in case of needing to migrations in the future.
2022-11-22 14:02:02 +00:00
Athanasius
424d5f023c
EDDNReplay.add_message() is now functional
And that includes the code to handle legacy `replay.json` messages.
2022-11-22 14:02:01 +00:00
Athanasius
03e432034f
EDDN: Moving replay functionality into its own class 2022-11-22 14:02:00 +00:00