1
0
mirror of https://github.com/EDCD/EDMarketConnector.git synced 2025-04-14 08:17:13 +03:00

5088 Commits

Author SHA1 Message Date
Philipp Trulson
c8e00304fb
Fix URL for EDDN GitHub Link 2022-11-24 14:09:43 +01:00
Athanasius
32229217b2
eddn: Bail from sending queued messages if one failed
NB: This is failed *not* due to the message being 'bad' in some manner. It
will mean the Gateway timed out, refused connection etc.
2022-11-23 16:00:28 +00:00
Athanasius
0eb33e011b
PLUGINS.md: Document new IsDocked state flag. 2022-11-23 15:57:25 +00:00
Athanasius
95fa9d577c
eddn: Use a shorter, 10 second, delay at startup for first queue check 2022-11-23 14:20:24 +00:00
Athanasius
3d9bb64330
eddn: Remove outdated comment about replaylog rework
And using tk `parent` *is* absolutely necessary.
2022-11-23 14:12:43 +00:00
Athanasius
b7769821be
eddn: Send immediately if docked
This has the "are we delaying sending?" check, but without the short-circuit
for "we are actually docked right now".
2022-11-23 13:56:01 +00:00
Athanasius
524c0425fb
eddn: EDDN.send_message(): Only set standard header if not already set.
Else we risk overwriting e.g. `CAPI-shipyard`.
2022-11-23 13:46:49 +00:00
Athanasius
fda91df04f
eddn: Working with tk after(), on timer or when docked
* An aborted attempt was made to use a thread worker, but:
  1. sqlite3 doesn't allow cross-thread use of the same sqlite3 connection.
  2. Having an on-going query on one cursor, e.g. gathering all the
    outstanding message `id`, whilst trying to DELETE a row hits a
    "database is locked" error.
* So, back to tk `after()`.  `send_message_by_id()` has been audited to ensure
  its boolean return is accurate.  So there shouldn't be any way in which to
  get hung up on a single message *other than if the EDDN Gateway is having
  issues, and thus it should be retried anyway*.  Any reason for a 'bad
  message' will cause `True` return and thus deletion of the message in
  *this* call to `queue_check_and_send()`.
* There is a new `reschedule` parameter to `queue_check_and_send()`.  If
  `True` then at the end it should re-schedule.

  There is a check in `journal_entry()` for the `Docked` event, and if this
  occurs it will schedule `queue_check_and_send()` with `reschedule` set to
  `False` so that we don't end up with multiple parallel schedulings.

  It's still possible for a docking to have coincided with a scheduled run
  and thus cause double-rate sending to EDDN, but we can live with that.

* The same scheduling mechanism is used, with a much smaller delay, to
  process more than one queued message per run.

  Hence the `have_rescheduled` bool *in* the function to indicate if a 'fast'
  reschedule has already been set.  This prevents the slow one *also* being
  set in this scenario.  The latter will be scheduled when the fast one
  found no more rows to process.
2022-11-23 13:29:47 +00:00
Athanasius
f2dbfacf70
eddn: Add header for CAPI outfitting and shipyard exports 2022-11-22 17:40:33 +00:00
Athanasius
f6e2504216
eddn: Fix capi_fcmaterials gameversion/build & add to CAPI commodity
* Erroneously used 'CAPI-commoodity' when it's 'CAPI-market' (name of the
  CAPI endpoint, not anything to do with EDDN schema names, and '-commodity'
  would also be wrong for that).
* Set `header` for (CAPI) `export_commodities()`.
2022-11-22 17:37:15 +00:00
Athanasius
b31c8c0536
eddn: A start on ensuring header (gameversion/build) are always set 2022-11-22 17:35:17 +00:00
Athanasius
4a0518da9f
eddn: Set header up properly for all journal messages
* New function `EDDN.add_header()`.
* It utilises new `this` members, set from `journal_entry()`.
2022-11-22 17:26:32 +00:00
dependabot[bot]
cee29ecb22
build(deps-dev): bump safety from 2.3.1 to 2.3.2
Bumps [safety](https://github.com/pyupio/safety) from 2.3.1 to 2.3.2.
- [Release notes](https://github.com/pyupio/safety/releases)
- [Changelog](https://github.com/pyupio/safety/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pyupio/safety/compare/2.3.1...2.3.2)

---
updated-dependencies:
- dependency-name: safety
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-22 17:04:24 +00:00
Athanasius
06fa3629ea
eddn: OUT_EDDN_DELAY (not inverted) & further work on the sending
* The eddn parts of the OUT_EDDN_DO_NOT_DELAY -> OUT_EDDN_DELAY change.  This
  includes the 'sense' of it being inverted from what it was.
* EDDN.REPLAY_DELAY is now a float, as it's used with `time.sleep()`.  *This*
  is the 400ms value for inter-message cooldown.
* EDDN.REPLAY_PERIOD is still an int, used with tk `after()`.  This is how
  often we attempt the queue.
* EDDN.session is no longer a thing, move that part of EDDN.close() to
  EDDNSender.close().
* EDDN queue DB has `id`, not `message_id`.
* Now *looping* in the queue sender, not only sending the oldest message.
2022-11-22 16:49:02 +00:00
Athanasius
6070f82c6b
OUT_EDDN_DO_NOT_DELAY should be OUT_EDDN_DELAY
* The old name was OUT_SYS_DELAY.
* Yes, this is the inverse of what we want, which is "should we not delay
  messages", but this is the legacy.
2022-11-22 16:47:17 +00:00
Athanasius
59f046ee67
eddn: First cut of periodic retry of sending messages 2022-11-22 15:51:10 +00:00
Athanasius
be1ef32238
config: Minor ordering change to ensure OUT_STATION_ANY set correctly
* This depends on `OUT_EDDN_SEND_STATION_DATA` which is defined below where
  this originally was.
2022-11-22 15:30:30 +00:00
Athanasius
a2d2723f49
monitor: Add an 'IsDocked' flag to monitor.state.
* This is cleaner than starting to track it in `plugins/eddn.py` specifically.
* This is literally only about if we're piloting a ship that is docked, so
  not even trying to resolve "on-foot, in a station, 'Location' said not
  docked though".
2022-11-22 15:29:05 +00:00
Athanasius
876c34ecfd
eddn: Remove two 'noqa: CCR001' that are no longer applicable 2022-11-22 14:09:08 +00:00
Athanasius
871f50288e
prefs.py: Fix overly long line (config constants renames) 2022-11-22 14:02:22 +00:00
Athanasius
06edcf3ea9
EDDN: Remove EDDN.send() in favour of renamed .export_journal_entry()
* Now that we're not trying to do "did we just/are we know docked?" in this
  code it turns out that both CAPI and Journal messages can use the same
  function for this.
* And as it's no longer journal-specific `EDDN.export_journal_entry()` has
  been renamed to `EDDN.send_message()`.

This whole branch now needs to actually implement sending queued messages
when docked, and periodically in the case of initial failures.
2022-11-22 14:02:21 +00:00
Athanasius
fe24cf7e95
EDDN: Remove legacy sendreplay()
* `EDDN.sendreplay()` is no longer used.
* In `prefsvarschanged()` there was a reference to `eddn.replayfile`, so as
  to grey out the "Delay sending..." option if the file wasn't available.
  So that's moot and also removed, but also...
* Comment the purpose of that line in `prefsvarchanged()` because it's not
  immediately obvious.
2022-11-22 14:02:20 +00:00
Athanasius
db7bb735d2
EDDN: Put "should we send (now?)" checks into EDDN.export_journal_entry()
In some cases the check might already have been done, but if not then this
is the last easy place to perform it.

NB: Unlike the old code this does *not* attempt to check "are we docked
  *now* ?" for triggering sending of previously queue messages.  That's
  going to need a thread worker.
2022-11-22 14:02:19 +00:00
Athanasius
03b36cbe39
EDDN: Improve export_journal_fcmaterials "don't send" comment 2022-11-22 14:02:18 +00:00
Athanasius
5eb4296ec6
EDDN: First steps to ensure new code respects user configuration
It's easier to check "should we send this message at all?" earlier. Currently
all of the following ('station data') do so:

* CAPI commodity, outfitting (also fcmaterials) and shipyard.
* Journal commodity, fcmaterials, outfitting,  and shipyard.
2022-11-22 14:02:17 +00:00
Athanasius
7d0ae88757
EDDN: EDDNSender.send_message() doesn't care about do/don't send options
By this point other code will have made that decision.
2022-11-22 14:02:16 +00:00
Athanasius
0d35f8874a
config/EDDN: Rename OUT_MKT_EDDN to OUT_EDDN_SEND_STATION_DATA
This flag controls whether commodity, outfitting or shipyard schema messages
are sent.  Thus 'MKT' ('market') is misleading.  Rename it so the intent when
used is clear.
2022-11-22 14:02:15 +00:00
Athanasius
9f02f18408
config/EDDN: Rename OUT_SYS_EDDN to OUT_EDDN_SEND_NON_STATION
* This was perhaps originally meant for what the UI option says, i.e. "send
  system and scan data", but is actually being used for anything that is
  **NOT** 'station data' (even though *that* option has 'MKT' it includes
  outfitting and shipyard as well).

  So, just name this more sanely such that code using it is more obvious as
  to the actual intent.
2022-11-22 14:02:14 +00:00
Athanasius
3a57b53bbd
config/EDDN: Rename OUT_SYS_DELAY to OUT_EDDN_DO_NOT_DELAY
The sense of this `output` flag has been inverted (always?) for a long time.

1. I have the option "Delay sending until docked" showing as *off* in the UI.
2. My config.output value is `100000000001`.
3. The value of this flag is `4096`, which means 12th bit (starting from 1, not
  zero).
4. So I have the bit set, but the option visibly off.

So, rename this both to be more pertinent to its use *and* to be correct as to
what `True` for it means.
2022-11-22 14:02:13 +00:00
Athanasius
598e54eaa4
EDDNSender: Now properly sends messages to Gateway
Including removing from the queue if it succeeded, or didn't and should be
dropped.
2022-11-22 14:02:12 +00:00
Athanasius
f66a98464e
EDDNSender: Closer to actually sending messages now 2022-11-22 14:02:11 +00:00
Athanasius
2b957d140c
EDDNSender: convert_legacy_file() belongs with "open the database" 2022-11-22 14:02:10 +00:00
Athanasius
0e20f4bc00
EDDNSender: Remove legacy file after migration 2022-11-22 14:02:09 +00:00
Athanasius
f3017d40ec
EDDNSender: Fill out type hints 2022-11-22 14:02:08 +00:00
Athanasius
51fb90b999
EDDN: Change EDDN.close() to call into EDDNSender.close() 2022-11-22 14:02:07 +00:00
Athanasius
c1793ad839
EDDN: Remove EDDN->flush() 2022-11-22 14:02:06 +00:00
Athanasius
09f646a249
EDDNSender: Add delete_message() method
This was tested by temporary code in `convert_legacy_file()` to delete
the last added row once all done.
2022-11-22 14:02:06 +00:00
Athanasius
86ff787aed
EDDNSender: Convert all of a legacy file
* I had a `break` in there to only convert the first message.
* Also collapsed the assignment to `cmdr, msg` to not go via `j`.
2022-11-22 14:02:05 +00:00
Athanasius
089c33002c
EDDNSender->add_message() returns ID of INSERTed row 2022-11-22 14:02:04 +00:00
Athanasius
80129361fe
EDDN: Refactor queue db open/creation to own function 2022-11-22 14:02:03 +00:00
Athanasius
9a660b3b49
EDDN: New class as EDDNSender now, and version the queue database file
* It makes more sense for this new class to be concerned with all the 'send it'
  functionality, not just 'replay', so rename it.
* Athough we're trying to get the schema right *first* time, let's plan ahead
  and version the filename in case of needing to migrations in the future.
2022-11-22 14:02:02 +00:00
Athanasius
424d5f023c
EDDNReplay.add_message() is now functional
And that includes the code to handle legacy `replay.json` messages.
2022-11-22 14:02:01 +00:00
Athanasius
03e432034f
EDDN: Moving replay functionality into its own class 2022-11-22 14:02:00 +00:00
Athanasius
072eadd893
EDDN: messages.id AUTOINCREMENT, and index created & cmdr
We'll definitely want to query against `cmdr`, and possibly `created`.

We shouldn't need to against other fields, they'll just be checked during
processing of an already selected message.
2022-11-22 14:01:59 +00:00
Athanasius
9faae8b9bc
EDDN: Open & create sqlite3 db for replay
* sqlite3 open, and creation of table.
* Change `load_journal_replay()` to `load_journal_replay_file()` and change
  the semantics to just return the `list[str]` loaded from it.  It also now
  catches no exceptions.
* Remove the "lock the journal cache" on init as it's not necessary.

There's still a lot more changes to come on this.
2022-11-22 14:01:58 +00:00
Athanasius
4c3ea5fd27
Merge pull request #1706 from EDCD/dependabot/pip/develop/setuptools-65.6.0
build(deps-dev): bump setuptools from 65.5.1 to 65.6.0
2022-11-22 12:17:44 +00:00
Athanasius
c9edfe358b
Merge pull request #1707 from EDCD/dependabot/pip/develop/flake8-isort-5.0.3
build(deps-dev): bump flake8-isort from 5.0.0 to 5.0.3
2022-11-22 12:17:34 +00:00
Athanasius
c3e3e6a0d0
Merge pull request #1702 from EDCD/dependabot/pip/develop/types-requests-2.28.11.5
build(deps-dev): bump types-requests from 2.28.11.2 to 2.28.11.5
2022-11-22 12:17:20 +00:00
dependabot[bot]
3ce5418bc2
build(deps-dev): bump flake8-isort from 5.0.0 to 5.0.3
Bumps [flake8-isort](https://github.com/gforcada/flake8-isort) from 5.0.0 to 5.0.3.
- [Release notes](https://github.com/gforcada/flake8-isort/releases)
- [Changelog](https://github.com/gforcada/flake8-isort/blob/master/CHANGES.rst)
- [Commits](https://github.com/gforcada/flake8-isort/compare/5.0.0...5.0.3)

---
updated-dependencies:
- dependency-name: flake8-isort
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-21 17:04:33 +00:00
dependabot[bot]
4f0a6ee12d
build(deps-dev): bump setuptools from 65.5.1 to 65.6.0
Bumps [setuptools](https://github.com/pypa/setuptools) from 65.5.1 to 65.6.0.
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/CHANGES.rst)
- [Commits](https://github.com/pypa/setuptools/compare/v65.5.1...v65.6.0)

---
updated-dependencies:
- dependency-name: setuptools
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-21 17:04:28 +00:00