From b35b934d8cf414c58127f926ad9bf9dc93192245 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Fri, 29 Oct 2021 13:42:43 +0100 Subject: [PATCH 01/26] schemas: blackmarket: README Note that it turns out this is deprecated in favour of the `prohibited` array defined in the `commodity` schema, and has been that way since September 2017. --- schemas/blackmarket-README.md | 53 +++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 schemas/blackmarket-README.md diff --git a/schemas/blackmarket-README.md b/schemas/blackmarket-README.md new file mode 100644 index 0000000..d278d27 --- /dev/null +++ b/schemas/blackmarket-README.md @@ -0,0 +1,53 @@ +# EDDN BlackMarket Schema + +## Introduction + +**This schema is deprecated. The same data has been made available via the +`prohibited` array in the [commodity schema](./commodity-v3.0.json) since +September 2017.** + +What follows below is only for historical curiosity. + +--- + +Here we document how to take data from an ED `BlackMArket` Journal Event and +properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event `MarketSell`. + +### Key Renames +#### name +Due to how the EDDN schema is defined the `Type` key/value should +have the key renamed to `name`. + +#### prohibited +Due to how the EDDN schema is defined the `IllegalGoods` key/value should +have the key renamed to `prohibited`. + +#### marketID +The Journal documentation says this is `MarketID`, but in the schema the +`m` is lower case. + +### Elisions +You MUST remove the following key/value pairs from the data: + + - `Count` + - `TotalSale` + - `AvgPricePaid` + - `StolenGoods` + - `BlackMarket` - Because we're using this schema, so this is un-necessary. + +### Augmentations +#### systemName +The star system name for where this market is. Use the `StarSystem` value +from the prior `Docked` or `Location` event. + +#### stationName +From the `StationName` value on the prior `Docked` or `Location` event. + +#### timestamp +An additional timsestamp, presumably from the `MarketSell` event. \ No newline at end of file From ed30aa4f01dbf91adf32b01f58909296afaf5869 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Fri, 29 Oct 2021 13:54:39 +0100 Subject: [PATCH 02/26] schemas: blackmarket: No, that's no an additional `timestamp` --- schemas/blackmarket-README.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/schemas/blackmarket-README.md b/schemas/blackmarket-README.md index d278d27..7193b1d 100644 --- a/schemas/blackmarket-README.md +++ b/schemas/blackmarket-README.md @@ -47,7 +47,4 @@ The star system name for where this market is. Use the `StarSystem` value from the prior `Docked` or `Location` event. #### stationName -From the `StationName` value on the prior `Docked` or `Location` event. - -#### timestamp -An additional timsestamp, presumably from the `MarketSell` event. \ No newline at end of file +From the `StationName` value on the prior `Docked` or `Location` event. \ No newline at end of file From 68160fcb8d0e7c6580b2d4182d432784cdccb240 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Fri, 29 Oct 2021 14:10:49 +0100 Subject: [PATCH 03/26] schemas: commodity: Initial README --- schemas/commodity-README.md | 46 +++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 schemas/commodity-README.md diff --git a/schemas/commodity-README.md b/schemas/commodity-README.md new file mode 100644 index 0000000..7941c36 --- /dev/null +++ b/schemas/commodity-README.md @@ -0,0 +1,46 @@ +# EDDN BlackMarket Schema + +## Introduction +Here we document how to take data from an ED `Market` Journal Event and +properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event `Market`, +and the additional file, `Market.json`, that it signals the writing of. + +So, look for the `Market` event, and when it occurs open and read the +`Market.json` file which contains the actual data. Treat *that* data as +the event. + +It *is* also possible to construct this data from a CAPI `/market` query. + +### Key Renames +Many of the key names have a different case defined in this schema, make +sure you are renaming them as appropriate. + +#### StarSystem to systemName +Rename the `StarSystem` key name to `systemName`. + +### Elisions +#### Remove _Localised key/values +You MUST remove the following key/value pairs from the data: + + - `StationType` key/value. + - All keys whose name ends with `_Localised`, i.e. the `Name_Localised` + key/values in Items. + - `Producer` key/value pair in Items. + - `Rare` key/value pair in Items. + - `id` key/value pair in Items. + +#### Item Category +Remove not only the `Category_Localised` key/value, as above, but also the +`Category` key/value pair from each Item. + +### Augmentations +#### horizons flag +Use the value from the `LoadGame` event. +#### odyssey flag +Use the value from the `LoadGame` event. From 8e03a8253612b597333f452f2d4e130dde73e0a6 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Fri, 29 Oct 2021 14:27:40 +0100 Subject: [PATCH 04/26] schemas: commodity: README: Document use of CAPI data --- schemas/commodity-README.md | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/schemas/commodity-README.md b/schemas/commodity-README.md index 7941c36..290b814 100644 --- a/schemas/commodity-README.md +++ b/schemas/commodity-README.md @@ -15,7 +15,8 @@ So, look for the `Market` event, and when it occurs open and read the `Market.json` file which contains the actual data. Treat *that* data as the event. -It *is* also possible to construct this data from a CAPI `/market` query. +It *is* also possible to construct this data from a CAPI `/market` query. +See [Using CAPI data](#using-capi-data) below. ### Key Renames Many of the key names have a different case defined in this schema, make @@ -44,3 +45,36 @@ Remove not only the `Category_Localised` key/value, as above, but also the Use the value from the `LoadGame` event. #### odyssey flag Use the value from the `LoadGame` event. + +### Using CAPI data +It is *not* recommended to use CAPI data as the source as it's fraught with +additional issues. EDMarketConnector does so in order to facilitate +obtaining data without the player needing to open the commodities screen. + +1. Retrieve the commander data from the `/profile` CAPI endpoint. +2. Check that `commander['docked']` is true. If not, abort. +3. Retrieve the data from the `/market` and `/shipyard` CAPI endpoints. +4. Compare the system and station name from the CAPI market data to that + from the last `Docked` or `Location` journal event. If either does not + match then you MUST **abort**. This likely indicates that the CAPI data is + lagging behind the game client state and thus should not be used. + +Note that CAPI `/market` data will sometimes have the `StatusFlasg` per +item, which are defined as optional in this schema (because they're not in +the Market.json data). You SHOULD include this data in your message if +using CAPI as the source. + +Now you will need to construct the necessary additional fields: + +#### CAPI horizons flag +You will need to check if any of the economies from the `/market` data have +a `name` are `Colony`, if so, set this flag true. + +Additionally, you should retrieve the CAPI `/shipyard` endpoint and check if +any of the listed ships or modules have a `sku` value of +`ELITE_HORIZONS_V_PLANETARY_LANDINGS`. If so, set this flag true. + +#### CAPI odyssey flag +Unfortunately there is no method to be *certain* of this from CAPI data, so +you will have to trust in the system/station name check and use the value +from the Journal `LoadGame` event. \ No newline at end of file From 8178e61b0097abf355b4896aeb489684c9402d63 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Fri, 29 Oct 2021 15:01:02 +0100 Subject: [PATCH 05/26] schemas: commodity: README: Correct top-level header --- schemas/commodity-README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/schemas/commodity-README.md b/schemas/commodity-README.md index 290b814..bd5721b 100644 --- a/schemas/commodity-README.md +++ b/schemas/commodity-README.md @@ -1,4 +1,4 @@ -# EDDN BlackMarket Schema +# EDDN Commodity Schema ## Introduction Here we document how to take data from an ED `Market` Journal Event and From d781160e6ec9879996759be98c07d8cb16908960 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Fri, 29 Oct 2021 15:05:28 +0100 Subject: [PATCH 06/26] schemas: commodity: README: _Localised keys correction --- schemas/commodity-README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/schemas/commodity-README.md b/schemas/commodity-README.md index bd5721b..31befef 100644 --- a/schemas/commodity-README.md +++ b/schemas/commodity-README.md @@ -27,11 +27,13 @@ Rename the `StarSystem` key name to `systemName`. ### Elisions #### Remove _Localised key/values +All keys whose name ends with `_Localised`, i.e. the `Name_Localised` +key/values in Items. + +#### Other Elisions You MUST remove the following key/value pairs from the data: - - `StationType` key/value. - - All keys whose name ends with `_Localised`, i.e. the `Name_Localised` - key/values in Items. +- `StationType` key/value. - `Producer` key/value pair in Items. - `Rare` key/value pair in Items. - `id` key/value pair in Items. From fc7443b2826d36929f569c886d35211f3b9e24d4 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Fri, 29 Oct 2021 15:09:30 +0100 Subject: [PATCH 07/26] schemas: fssdiscoveryscan: README: Initial version --- schemas/fssdiscoveryscan-README.md | 40 ++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 schemas/fssdiscoveryscan-README.md diff --git a/schemas/fssdiscoveryscan-README.md b/schemas/fssdiscoveryscan-README.md new file mode 100644 index 0000000..594adc4 --- /dev/null +++ b/schemas/fssdiscoveryscan-README.md @@ -0,0 +1,40 @@ +# EDDN FSSDiscoveryScan Schema + +## Introduction +Here we document how to take data from an ED `FSSDiscoveryScan` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event +`FSSDiscoveryScan`. + +### Key Renames +Many of the key names have a different case defined in this schema, make +sure you are renaming them as appropriate. + +### Elisions +#### Remove _Localised key/values +All keys whose name ends with `_Localised`, i.e. the `Name_Localised` +key/values in Items. + +#### Other Elisions +You MUST remove the following key/value pairs from the data: + + - `Progress` key/value pair. + +#### Item Category +Remove not only the `Category_Localised` key/value, as above, but also the +`Category` key/value pair from each Item. + +### Augmentations +#### horizons flag +Use the value from the `LoadGame` event. +#### odyssey flag +Use the value from the `LoadGame` event. + +#### StarPos +You MUST add a `StarPos` array containing the system co-ordinates from the +last `FSDJump`, `CarrierJump`, or `Location` event. \ No newline at end of file From 635d7c269375531d175b952f89c4d7c6804c68fa Mon Sep 17 00:00:00 2001 From: Athanasius Date: Mon, 1 Nov 2021 14:31:59 +0000 Subject: [PATCH 08/26] schemas: README: Note about key names being originally CAPI based --- schemas/README-EDDN-schemas.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/schemas/README-EDDN-schemas.md b/schemas/README-EDDN-schemas.md index 550a56e..8ce4be2 100644 --- a/schemas/README-EDDN-schemas.md +++ b/schemas/README-EDDN-schemas.md @@ -170,6 +170,10 @@ Each `message` object must have, at bare minimum: much more than this. Again, consult the [schemas and their documentation](./). +Note that many of the key names chosen in the schemas are based on the CAPI +data, not Journal events, because the CAPI came first. This means renaming +many of the keys from Journal events to match the schema. + EDDN is intended to transport generic data not specific to any particular Cmdr and to reflect the data that a player would see in-game in station services or the local map. To that end, uploading applications MUST ensure that messages do From fd567e69aad6b754d9812d758f72abd92a7141cb Mon Sep 17 00:00:00 2001 From: Athanasius Date: Mon, 1 Nov 2021 14:44:28 +0000 Subject: [PATCH 09/26] schemas: journal: README: Now complete, in the generic sense Ideally all the events covered here need splitting into their own schemas, but for the time being this documentation should suffice. --- schemas/journal-README.md | 86 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 schemas/journal-README.md diff --git a/schemas/journal-README.md b/schemas/journal-README.md new file mode 100644 index 0000000..5dd7cea --- /dev/null +++ b/schemas/journal-README.md @@ -0,0 +1,86 @@ +# EDDN Journal Schema + +## Introduction +Here we document how to take data from miscellaneous ED Journal +events and properly structure it for sending to EDDN. + +This is the historical "all Journal events" schema that will be deprecated +in the future. Please check for a schema specific to the journal event +under consideration to see if data should be sent on that event specific +schema instead. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal events: + + - `Docked` + - `FSDJump` + - `Scan` + - `Location` + - `SAASignalsFound` + - `CarrierJump` + - `CodexEntry` - But see the separate + [codexentry schema](./codexentry-README.md) documentation. + +### Key Renames +Many of the key names have a different case defined in this schema, make +sure you are renaming them as appropriate. + +### Elisions +#### Remove _Localised key/values +All keys whose name ends with `_Localised`, i.e. the `Name_Localised` +key/values in Items. + +#### Personal data in `Docked` events +The following keys+values should be removed from `Docked` event data: + + - `Wanted` + - `ActiveFine` + - `CockpitBreach` + +#### Personal data in `FSDJump` events +The following keys+values should be removed from `FSDJump` event data: + +- `Wanted` +- `BoostUsed` +- `FuelLevel` +- `FuelUsed` +- `JumpDist` +- `HappiestSystem` from within the list of `Factions`. +- `HomeSystem` from within the list of `Factions`. +- `MyReputation` from within the list of `Factions`. +- `SquadronFaction` from within the list of `Factions`. + +#### Personal data in `Location` events +The following keys+values should be removed from `Location` event data: + +- `Wanted` +- `Latitude` +- `Longitude` +- `HappiestSystem` from within the list of `Factions`. +- `HomeSystem` from within the list of `Factions`. +- `MyReputation` from within the list of `Factions`. +- `SquadronFaction` from within the list of `Factions`. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### StarSystem +You MUST add a `StarSystem` key/value pair representing the name of the +system this event occurred in. Source this from either `Location`, +`FSDJump` or `CarrierJump` as appropriate. + +#### SystemAddress +You MUST add a `SystemAddress` key/value pair representing the numerical ID +of the system this event occurred in. Source this from either `Location`, +`FSDJump` or `CarrierJump` as appropriate. + +#### StarPos +You MUST add a `StarPos` array containing the system co-ordinates from the +last `FSDJump`, `CarrierJump`, or `Location` event. From 67e77136ffd0a8e4cfebbea71882cb3980cc5096 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Mon, 1 Nov 2021 14:54:10 +0000 Subject: [PATCH 10/26] schemas: navbeaconscan: README: Initial version --- schemas/navbeaconscan-README.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 schemas/navbeaconscan-README.md diff --git a/schemas/navbeaconscan-README.md b/schemas/navbeaconscan-README.md new file mode 100644 index 0000000..c12e91b --- /dev/null +++ b/schemas/navbeaconscan-README.md @@ -0,0 +1,31 @@ +# EDDN NavBeaconScan Schema + +## Introduction +Here we document how to take data from an ED `NavBeaconScan` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event +`NavBeaconScan`. + +### Elisions +There are no elisions in this schema. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### StarSystem +You MUST add a `StarSystem` key/value pair representing the name of the +system this event occurred in. Source this from either `Location`, +`FSDJump` or `CarrierJump` as appropriate. + +#### StarPos +You MUST add a `StarPos` array containing the system co-ordinates from the +last `FSDJump`, `CarrierJump`, or `Location` event. From 6c1d8cff92c745e7fd98dea9e48a74b53f9fa78c Mon Sep 17 00:00:00 2001 From: Athanasius Date: Mon, 1 Nov 2021 14:58:54 +0000 Subject: [PATCH 11/26] schemas: navroute: README: Initial version --- schemas/navroute-README.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 schemas/navroute-README.md diff --git a/schemas/navroute-README.md b/schemas/navroute-README.md new file mode 100644 index 0000000..3025baa --- /dev/null +++ b/schemas/navroute-README.md @@ -0,0 +1,29 @@ +# EDDN NavRoute Schema + +## Introduction +Here we document how to take data from an ED `NavRoute` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the `NavRoute.json` file. That +it has been freshly written is signalled by the ED Journal event `NavRoute`. + +So, monitor the Journal as normal, and when you see a `NavRoute` event open +the `NavRoute.json` file for reading, read it, and close it again. Use the +data you got from reading this file, not merely the Journal event. + +The primary data to be sent is the `Route` array from the contents of the +separate file. + +### Elisions +There are no elisions in this schema. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. \ No newline at end of file From a32d6e2ea337650bdc18b8a3c810e0939a04337a Mon Sep 17 00:00:00 2001 From: Athanasius Date: Mon, 1 Nov 2021 15:11:15 +0000 Subject: [PATCH 12/26] schemas: outfitting: README: Initial version --- schemas/outfitting-README.md | 41 ++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 schemas/outfitting-README.md diff --git a/schemas/outfitting-README.md b/schemas/outfitting-README.md new file mode 100644 index 0000000..f3e4370 --- /dev/null +++ b/schemas/outfitting-README.md @@ -0,0 +1,41 @@ +# EDDN Outfitting Schema + +## Introduction +Here we document how to take data from an ED `Outfitting` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event +`Outfitting`. + +You MAY also source this data from the CAPI `/shipyard` endpoint. See +[commodity-README.md#using-capi-data](commodity-README.md#using-capi-data) +for guidance on this. + +You only need the `name` key's value for each member of the `modules` array. + +### Key Renames +Many of the key names have a different case defined in this schema, make +sure you are renaming them as appropriate. + +### Elisions +Remove items whose availability depends on the Cmdr's status rather than on the +station. Namely: + +- Items that aren't weapons/utilities (`Hpt_*`), standard/internal + modules (`Int_*`) or armour (`*_Armour_*`) (i.e. bobbleheads, decals, + paintjobs and shipkits). This is enforced by the schema. +- Items that have a non-null `"sku"` property, unless + it's `"ELITE_HORIZONS_V_PLANETARY_LANDINGS"` (i.e. PowerPlay and tech + broker items). +- The `"Int_PlanetApproachSuite"` module (for historical reasons). + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. \ No newline at end of file From e12ebee9f8f370bc456a57eaef5d63c1229fc599 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Mon, 1 Nov 2021 15:15:03 +0000 Subject: [PATCH 13/26] schemas: scanbarycentre: README: Initial version --- schemas/scanbarycentre-README.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 schemas/scanbarycentre-README.md diff --git a/schemas/scanbarycentre-README.md b/schemas/scanbarycentre-README.md new file mode 100644 index 0000000..da0e178 --- /dev/null +++ b/schemas/scanbarycentre-README.md @@ -0,0 +1,26 @@ +# EDDN ScanBaryCentre Schema + +## Introduction +Here we document how to take data from an ED `ScanBaryCentre` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event +`ScanBaryCentre`. + +### Elisions +There are no elisions in this schema. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### StarPos +You MUST add a `StarPos` array containing the system co-ordinates from the +last `FSDJump`, `CarrierJump`, or `Location` event. From 3d023bf2801b8c6660328351ba60ba05d43693f6 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Mon, 1 Nov 2021 15:46:08 +0000 Subject: [PATCH 14/26] schemas: shipyard: README: Initial version --- schemas/shipyard-README.md | 39 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 schemas/shipyard-README.md diff --git a/schemas/shipyard-README.md b/schemas/shipyard-README.md new file mode 100644 index 0000000..295ff5a --- /dev/null +++ b/schemas/shipyard-README.md @@ -0,0 +1,39 @@ +# EDDN Shipyard Schema + +## Introduction +Here we document how to take data from an ED `Shipyard` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event +`Shipyard`. + +You MAY also source this data from the CAPI `/shipyard` endpoint. See +[commodity-README.md#using-capi-data](commodity-README.md#using-capi-data) +for guidance on this. + + You only need the `name` key's value for each member of the `PriceList` +array (if using Journal, it will be from the `ships` array if using CAPI +data). + +When using CAPI data *include* ships listed in the `"unavailable_list"` +property (i.e. available at this station, but not to this Cmdr). + +This list of ship names will go in the `ships` array in the EDDN message. + +### Key Renames +Many of the key names have a different case defined in this schema, make +sure you are renaming them as appropriate. + +### Elisions +There are no elisions in this schema. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. From 339abce9e45591c2938a924e93a54d7551d27194 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Mon, 1 Nov 2021 15:52:56 +0000 Subject: [PATCH 15/26] schemas: READMES: Consistent horizons/odyssey language --- schemas/blackmarket-README.md | 6 ++++++ schemas/codexentry-README.md | 6 ++++++ schemas/commodity-README.md | 5 +++-- schemas/fssdiscoveryscan-README.md | 5 +++-- 4 files changed, 18 insertions(+), 4 deletions(-) diff --git a/schemas/blackmarket-README.md b/schemas/blackmarket-README.md index 7193b1d..c86035e 100644 --- a/schemas/blackmarket-README.md +++ b/schemas/blackmarket-README.md @@ -42,6 +42,12 @@ You MUST remove the following key/value pairs from the data: - `BlackMarket` - Because we're using this schema, so this is un-necessary. ### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + #### systemName The star system name for where this market is. Use the `StarSystem` value from the prior `Docked` or `Location` event. diff --git a/schemas/codexentry-README.md b/schemas/codexentry-README.md index 69b6e4a..1f3725e 100644 --- a/schemas/codexentry-README.md +++ b/schemas/codexentry-README.md @@ -17,6 +17,12 @@ You MUST remove any key where the key name ends in You MUST remove the two keys `IsNewEntry` and `NewTraitsDiscovered`. ### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + #### StarPos You MUST **add** a `StarPos` key with value of type `array` containing the galaxy co-ordinates of the system. You will need to have obtained these diff --git a/schemas/commodity-README.md b/schemas/commodity-README.md index 31befef..cdc9bc9 100644 --- a/schemas/commodity-README.md +++ b/schemas/commodity-README.md @@ -44,9 +44,10 @@ Remove not only the `Category_Localised` key/value, as above, but also the ### Augmentations #### horizons flag -Use the value from the `LoadGame` event. +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + #### odyssey flag -Use the value from the `LoadGame` event. +You SHOULD add this key/value pair, using the value from the `LoadGame` event. ### Using CAPI data It is *not* recommended to use CAPI data as the source as it's fraught with diff --git a/schemas/fssdiscoveryscan-README.md b/schemas/fssdiscoveryscan-README.md index 594adc4..de16148 100644 --- a/schemas/fssdiscoveryscan-README.md +++ b/schemas/fssdiscoveryscan-README.md @@ -31,9 +31,10 @@ Remove not only the `Category_Localised` key/value, as above, but also the ### Augmentations #### horizons flag -Use the value from the `LoadGame` event. +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + #### odyssey flag -Use the value from the `LoadGame` event. +You SHOULD add this key/value pair, using the value from the `LoadGame` event. #### StarPos You MUST add a `StarPos` array containing the system co-ordinates from the From b9231cef12432fa04e6480369c3a054afaae4544 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Mon, 1 Nov 2021 15:58:00 +0000 Subject: [PATCH 16/26] schemas: READMEs: Move commodity-specific elisions into its file --- schemas/README-EDDN-schemas.md | 3 --- schemas/commodity-README.md | 12 +++++++++--- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/schemas/README-EDDN-schemas.md b/schemas/README-EDDN-schemas.md index 8ce4be2..a86e1b5 100644 --- a/schemas/README-EDDN-schemas.md +++ b/schemas/README-EDDN-schemas.md @@ -180,9 +180,6 @@ the local map. To that end, uploading applications MUST ensure that messages do not contain any Cmdr-specific data (other than "uploaderID" and the "horizons" flag). In practice as of E:D 3.3 this means: -* commodity: Skip commodities with `"categoryname": "NonMarketable"` (i.e. - Limpets - not purchasable in station market) or `"legality":` *non-empty - string* (not normally traded at this station market). * outfitting: Skip items whose availability depends on the Cmdr's status rather than on the station. Namely: - Items that aren't weapons/utilities (`Hpt_*`), standard/internal diff --git a/schemas/commodity-README.md b/schemas/commodity-README.md index cdc9bc9..2452530 100644 --- a/schemas/commodity-README.md +++ b/schemas/commodity-README.md @@ -34,9 +34,15 @@ key/values in Items. You MUST remove the following key/value pairs from the data: - `StationType` key/value. - - `Producer` key/value pair in Items. - - `Rare` key/value pair in Items. - - `id` key/value pair in Items. +- `Producer` key/value pair in Items. +- `Rare` key/value pair in Items. +- `id` key/value pair in Items. + +In the list of commodites: + +- Skip commodities with `"categoryname": "NonMarketable"` (i.e. + Limpets - not purchasable in station market) or a *non-empty*`"legality":` + string (not normally traded at this station market). #### Item Category Remove not only the `Category_Localised` key/value, as above, but also the From a32f047b7f8dcf93526a934b822d7e9502e4ba59 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Mon, 1 Nov 2021 15:58:35 +0000 Subject: [PATCH 17/26] schemas: README: outfitting-specific elisions in that file already --- schemas/README-EDDN-schemas.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/schemas/README-EDDN-schemas.md b/schemas/README-EDDN-schemas.md index a86e1b5..a986f5f 100644 --- a/schemas/README-EDDN-schemas.md +++ b/schemas/README-EDDN-schemas.md @@ -180,15 +180,6 @@ the local map. To that end, uploading applications MUST ensure that messages do not contain any Cmdr-specific data (other than "uploaderID" and the "horizons" flag). In practice as of E:D 3.3 this means: -* outfitting: Skip items whose availability depends on the Cmdr's status rather - than on the station. Namely: - - Items that aren't weapons/utilities (`Hpt_*`), standard/internal - modules (`Int_*`) or armour (`*_Armour_*`) (i.e. bobbleheads, decals, - paintjobs and shipkits). - - Items that have a non-null `"sku"` property, unless - it's `"ELITE_HORIZONS_V_PLANETARY_LANDINGS"` (i.e. PowerPlay and tech - broker items). - - The `"Int_PlanetApproachSuite"` module (for historical reasons). * shipyard: *Include* ships listed in the `"unavailable_list"` property (i.e. available at this station, but not to this Cmdr). * journal: Strip out `"..._Localised"` properties throughout the data From 57e8b9af4b8d43e212966ed7dcd1a17783b7b494 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Mon, 1 Nov 2021 15:59:38 +0000 Subject: [PATCH 18/26] schemas: README: shipyard `unavailable_list` already noted in its README --- schemas/README-EDDN-schemas.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/schemas/README-EDDN-schemas.md b/schemas/README-EDDN-schemas.md index a986f5f..ae7f5ed 100644 --- a/schemas/README-EDDN-schemas.md +++ b/schemas/README-EDDN-schemas.md @@ -180,8 +180,6 @@ the local map. To that end, uploading applications MUST ensure that messages do not contain any Cmdr-specific data (other than "uploaderID" and the "horizons" flag). In practice as of E:D 3.3 this means: -* shipyard: *Include* ships listed in the `"unavailable_list"` property (i.e. - available at this station, but not to this Cmdr). * journal: Strip out `"..._Localised"` properties throughout the data structure. * journal/Docked: Strip out `"Wanted"`, `"ActiveFine"`, `"CockpitBreach"` From aebf3ac4794b036804d6a4d69e96a549d4b7629f Mon Sep 17 00:00:00 2001 From: Athanasius Date: Mon, 1 Nov 2021 16:02:12 +0000 Subject: [PATCH 19/26] schemas: README: Remove final things now mentioned in specific READMEs --- schemas/README-EDDN-schemas.md | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/schemas/README-EDDN-schemas.md b/schemas/README-EDDN-schemas.md index ae7f5ed..e4c8c6a 100644 --- a/schemas/README-EDDN-schemas.md +++ b/schemas/README-EDDN-schemas.md @@ -178,19 +178,10 @@ EDDN is intended to transport generic data not specific to any particular Cmdr and to reflect the data that a player would see in-game in station services or the local map. To that end, uploading applications MUST ensure that messages do not contain any Cmdr-specific data (other than "uploaderID" and the "horizons" -flag). In practice as of E:D 3.3 this means: +flag). -* journal: Strip out `"..._Localised"` properties throughout the data - structure. -* journal/Docked: Strip out `"Wanted"`, `"ActiveFine"`, `"CockpitBreach"` - properties -* journal/FSDJump: Strip out `"Wanted"`, `"BoostUsed"`, `"FuelLevel"` - , `"FuelUsed"` and `"JumpDist"` properties. -* journal/Location: Strip out `"Wanted"`, `"Latitude"` and `"Longitude"` - properties. -* journal/Location and journal/FSDJump: strip out `"HappiestSystem"` - , `"HomeSystem"`, `"MyReputation"` and `"SquadronFaction"` properties within - the list of `"Factions"`. +The individual schemas will instruct you on various elisions (removals) to +be made to comply with this. Some of these requirements are also enforced by the schemas, and some things the schemas enforce might not be explicitly called out here, so **do** From 53acd5ed1e5af0af740763870140e3206d07d96d Mon Sep 17 00:00:00 2001 From: Athanasius Date: Tue, 2 Nov 2021 11:20:09 +0000 Subject: [PATCH 20/26] schemas: blackmarket: README: Correct referenced Journal event --- schemas/blackmarket-README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/schemas/blackmarket-README.md b/schemas/blackmarket-README.md index c86035e..11d9c76 100644 --- a/schemas/blackmarket-README.md +++ b/schemas/blackmarket-README.md @@ -10,7 +10,7 @@ What follows below is only for historical curiosity. --- -Here we document how to take data from an ED `BlackMArket` Journal Event and +Here we document how to take data from an ED `MarketSell` Journal Event and properly structure it for sending to EDDN. Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general From 6e81afc98c439991fcaef6b525f77931510b6df0 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Tue, 2 Nov 2021 11:20:43 +0000 Subject: [PATCH 21/26] schemas: blackmarket: README: `Count` doesn't need to be elided --- schemas/blackmarket-README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/schemas/blackmarket-README.md b/schemas/blackmarket-README.md index 11d9c76..0054bd1 100644 --- a/schemas/blackmarket-README.md +++ b/schemas/blackmarket-README.md @@ -35,7 +35,6 @@ The Journal documentation says this is `MarketID`, but in the schema the ### Elisions You MUST remove the following key/value pairs from the data: - - `Count` - `TotalSale` - `AvgPricePaid` - `StolenGoods` From 5b05e4879f855e6f51423042cf4971ee16f864f6 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Tue, 2 Nov 2021 11:24:37 +0000 Subject: [PATCH 22/26] schemas: commodity: README: Call out `statusFlags` --- schemas/commodity-README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/schemas/commodity-README.md b/schemas/commodity-README.md index 2452530..e50d411 100644 --- a/schemas/commodity-README.md +++ b/schemas/commodity-README.md @@ -18,6 +18,11 @@ the event. It *is* also possible to construct this data from a CAPI `/market` query. See [Using CAPI data](#using-capi-data) below. +### Statusflags +`statusFlags` is an optional augmentation that can be used to identify +`Producer`, `Consumer`, and `Rare` goods from the output of `Market.json`. +So, as per the schema, do include it if available. + ### Key Renames Many of the key names have a different case defined in this schema, make sure you are renaming them as appropriate. From 6704e1a957e92509e1c2eeed59103b75619d80c2 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Tue, 2 Nov 2021 11:30:25 +0000 Subject: [PATCH 23/26] schemas: commodity: README: Allow `LoadGame` horizons flag for CAPI data --- schemas/commodity-README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/schemas/commodity-README.md b/schemas/commodity-README.md index e50d411..7abbedf 100644 --- a/schemas/commodity-README.md +++ b/schemas/commodity-README.md @@ -81,8 +81,12 @@ using CAPI as the source. Now you will need to construct the necessary additional fields: #### CAPI horizons flag -You will need to check if any of the economies from the `/market` data have -a `name` are `Colony`, if so, set this flag true. +If your application can be certain that the game client is still running, +and logged into the game (not just run to the main menu), then you can +simply use the value from the `LoadGame` journal event. + +Otherwise, you MUST check if any of the economies from the `/market` +data have a `name` of `Colony`, if so, set this flag true. Additionally, you should retrieve the CAPI `/shipyard` endpoint and check if any of the listed ships or modules have a `sku` value of From e0c1ac85fd3492fa6056f756b1474044dbb9b47a Mon Sep 17 00:00:00 2001 From: Athanasius Date: Tue, 2 Nov 2021 11:32:54 +0000 Subject: [PATCH 24/26] schemas: fssdiscoveryscan: README: Don't over-state elisions --- schemas/fssdiscoveryscan-README.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/schemas/fssdiscoveryscan-README.md b/schemas/fssdiscoveryscan-README.md index de16148..6d4d844 100644 --- a/schemas/fssdiscoveryscan-README.md +++ b/schemas/fssdiscoveryscan-README.md @@ -16,19 +16,10 @@ Many of the key names have a different case defined in this schema, make sure you are renaming them as appropriate. ### Elisions -#### Remove _Localised key/values -All keys whose name ends with `_Localised`, i.e. the `Name_Localised` -key/values in Items. - -#### Other Elisions You MUST remove the following key/value pairs from the data: - `Progress` key/value pair. -#### Item Category -Remove not only the `Category_Localised` key/value, as above, but also the -`Category` key/value pair from each Item. - ### Augmentations #### horizons flag You SHOULD add this key/value pair, using the value from the `LoadGame` event. From 1b11519ee7ebb7b47e3ba0bba36badd85933cd0a Mon Sep 17 00:00:00 2001 From: Athanasius Date: Thu, 4 Nov 2021 11:03:51 +0000 Subject: [PATCH 25/26] schemas: READMEs: Centralise "detecting CAPI lag" advice --- schemas/README-EDDN-schemas.md | 20 +++++++++++++++++--- schemas/commodity-README.md | 10 +++------- schemas/outfitting-README.md | 7 ++++--- schemas/shipyard-README.md | 7 ++++--- 4 files changed, 28 insertions(+), 16 deletions(-) diff --git a/schemas/README-EDDN-schemas.md b/schemas/README-EDDN-schemas.md index e4c8c6a..a7461d8 100644 --- a/schemas/README-EDDN-schemas.md +++ b/schemas/README-EDDN-schemas.md @@ -62,13 +62,27 @@ the available endpoints and how they work. There is some [third-party documentation](https://github.com/Athanasius/fd-api/blob/main/docs/README.md) by Athanasius. +It is *not* recommended to use CAPI data as the source as it's fraught with +additional issues. EDMarketConnector does so in order to facilitate +obtaining data without the player needing to open the commodities screen. + +#### Detecting CAPI data lag + When using the Companion API please be aware that the server that supplies this data sometimes lags behind the game - usually by a few seconds, sometimes by minutes. You MUST check in the data from the CAPI that the Cmdr is -docked (`["commander"]["docked"]` is `True`) and that the station and -system (`["lastStarport"]["name"]` and `["lastSystem"]["name"]`) match those +docked, and that the station and system names match those reported from the Journal before using the data for the commodity, outfitting -and shipyard schemas. +and shipyard schemas: + +1. Retrieve the commander data from the `/profile` CAPI endpoint. +2. Check that `commander['docked']` is true. If not, abort. +3. Retrieve the data from the `/market` and `/shipyard` CAPI endpoints. +4. Compare the system and station name from the CAPI market data, + `["lastStarport"]["name"]` and `["lastSystem"]["name"]`, + to that from the last `Docked` or `Location` journal event. If either does + not match then you MUST **abort**. This likely indicates that the CAPI + data is lagging behind the game client state and thus should not be used. --- diff --git a/schemas/commodity-README.md b/schemas/commodity-README.md index 7abbedf..c2a4516 100644 --- a/schemas/commodity-README.md +++ b/schemas/commodity-README.md @@ -65,13 +65,9 @@ It is *not* recommended to use CAPI data as the source as it's fraught with additional issues. EDMarketConnector does so in order to facilitate obtaining data without the player needing to open the commodities screen. -1. Retrieve the commander data from the `/profile` CAPI endpoint. -2. Check that `commander['docked']` is true. If not, abort. -3. Retrieve the data from the `/market` and `/shipyard` CAPI endpoints. -4. Compare the system and station name from the CAPI market data to that - from the last `Docked` or `Location` journal event. If either does not - match then you MUST **abort**. This likely indicates that the CAPI data is - lagging behind the game client state and thus should not be used. +Please read +[the guidance on checking for CAPI lag](README-EDDN-schemas.md#detecting-capi-data-lag) +before utilising CAPI data for EDDN messages. Note that CAPI `/market` data will sometimes have the `StatusFlasg` per item, which are defined as optional in this schema (because they're not in diff --git a/schemas/outfitting-README.md b/schemas/outfitting-README.md index f3e4370..090fc61 100644 --- a/schemas/outfitting-README.md +++ b/schemas/outfitting-README.md @@ -11,9 +11,10 @@ documentation for a schema such as this. The primary data source for this schema is the ED Journal event `Outfitting`. -You MAY also source this data from the CAPI `/shipyard` endpoint. See -[commodity-README.md#using-capi-data](commodity-README.md#using-capi-data) -for guidance on this. +You MAY also source this data from the CAPI `/shipyard` endpoint. +Please read +[the guidance on checking for CAPI lag](README-EDDN-schemas.md#detecting-capi-data-lag) +before utilising CAPI data for EDDN messages. You only need the `name` key's value for each member of the `modules` array. diff --git a/schemas/shipyard-README.md b/schemas/shipyard-README.md index 295ff5a..33016a4 100644 --- a/schemas/shipyard-README.md +++ b/schemas/shipyard-README.md @@ -11,9 +11,10 @@ documentation for a schema such as this. The primary data source for this schema is the ED Journal event `Shipyard`. -You MAY also source this data from the CAPI `/shipyard` endpoint. See -[commodity-README.md#using-capi-data](commodity-README.md#using-capi-data) -for guidance on this. +You MAY also source this data from the CAPI `/shipyard` endpoint. +Please read +[the guidance on checking for CAPI lag](README-EDDN-schemas.md#detecting-capi-data-lag) +before utilising CAPI data for EDDN messages. You only need the `name` key's value for each member of the `PriceList` array (if using Journal, it will be from the `ships` array if using CAPI From d6a43c04d9b086d62f09223c28ffa3bdafdcb9e5 Mon Sep 17 00:00:00 2001 From: Athanasius Date: Thu, 4 Nov 2021 11:14:47 +0000 Subject: [PATCH 26/26] schemas: README: Add 'Receiving messages' section --- schemas/README-EDDN-schemas.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/schemas/README-EDDN-schemas.md b/schemas/README-EDDN-schemas.md index a7461d8..83f6742 100644 --- a/schemas/README-EDDN-schemas.md +++ b/schemas/README-EDDN-schemas.md @@ -201,3 +201,29 @@ Some of these requirements are also enforced by the schemas, and some things the schemas enforce might not be explicitly called out here, so **do** check what you're sending against the schema when implementing sending new events. + +## Receiving messages + +EDDN provides a continuous stream of information from uploaders. To use this +data you'll need to connect to the stream using ZeroMQ (a library is probably +available for your language of choice). + +The URL for the live Relay is: + + tcp://eddn.edcd.io:9500 + +Once you've connected to that you will receive messages. To access the +data you will first need to zlib-decompress each message. Then you will +have a textual JSON object as per the schemas. + +In general, check the guidance for [Uploading messages](#uploading-messages) +for the expected format of the messages. + +Consumers can utilise the `$schemaRef` value to determine which schema a +particular message is for. There is no need to validate the messages +against the schemas yourself, as that is performed on the EDDN Gateway. +Messages that do not pass the schema validation there are not forwarded to +receivers. + +There is [example code](https://github.com/EDCD/EDDN/tree/master/examples) +available for a variety of programming languages to help you get started. \ No newline at end of file