diff --git a/schemas/README-EDDN-schemas.md b/schemas/README-EDDN-schemas.md index 550a56e..83f6742 100644 --- a/schemas/README-EDDN-schemas.md +++ b/schemas/README-EDDN-schemas.md @@ -62,13 +62,27 @@ the available endpoints and how they work. There is some [third-party documentation](https://github.com/Athanasius/fd-api/blob/main/docs/README.md) by Athanasius. +It is *not* recommended to use CAPI data as the source as it's fraught with +additional issues. EDMarketConnector does so in order to facilitate +obtaining data without the player needing to open the commodities screen. + +#### Detecting CAPI data lag + When using the Companion API please be aware that the server that supplies this data sometimes lags behind the game - usually by a few seconds, sometimes by minutes. You MUST check in the data from the CAPI that the Cmdr is -docked (`["commander"]["docked"]` is `True`) and that the station and -system (`["lastStarport"]["name"]` and `["lastSystem"]["name"]`) match those +docked, and that the station and system names match those reported from the Journal before using the data for the commodity, outfitting -and shipyard schemas. +and shipyard schemas: + +1. Retrieve the commander data from the `/profile` CAPI endpoint. +2. Check that `commander['docked']` is true. If not, abort. +3. Retrieve the data from the `/market` and `/shipyard` CAPI endpoints. +4. Compare the system and station name from the CAPI market data, + `["lastStarport"]["name"]` and `["lastSystem"]["name"]`, + to that from the last `Docked` or `Location` journal event. If either does + not match then you MUST **abort**. This likely indicates that the CAPI + data is lagging behind the game client state and thus should not be used. --- @@ -170,39 +184,46 @@ Each `message` object must have, at bare minimum: much more than this. Again, consult the [schemas and their documentation](./). +Note that many of the key names chosen in the schemas are based on the CAPI +data, not Journal events, because the CAPI came first. This means renaming +many of the keys from Journal events to match the schema. + EDDN is intended to transport generic data not specific to any particular Cmdr and to reflect the data that a player would see in-game in station services or the local map. To that end, uploading applications MUST ensure that messages do not contain any Cmdr-specific data (other than "uploaderID" and the "horizons" -flag). In practice as of E:D 3.3 this means: +flag). -* commodity: Skip commodities with `"categoryname": "NonMarketable"` (i.e. - Limpets - not purchasable in station market) or `"legality":` *non-empty - string* (not normally traded at this station market). -* outfitting: Skip items whose availability depends on the Cmdr's status rather - than on the station. Namely: - - Items that aren't weapons/utilities (`Hpt_*`), standard/internal - modules (`Int_*`) or armour (`*_Armour_*`) (i.e. bobbleheads, decals, - paintjobs and shipkits). - - Items that have a non-null `"sku"` property, unless - it's `"ELITE_HORIZONS_V_PLANETARY_LANDINGS"` (i.e. PowerPlay and tech - broker items). - - The `"Int_PlanetApproachSuite"` module (for historical reasons). -* shipyard: *Include* ships listed in the `"unavailable_list"` property (i.e. - available at this station, but not to this Cmdr). -* journal: Strip out `"..._Localised"` properties throughout the data - structure. -* journal/Docked: Strip out `"Wanted"`, `"ActiveFine"`, `"CockpitBreach"` - properties -* journal/FSDJump: Strip out `"Wanted"`, `"BoostUsed"`, `"FuelLevel"` - , `"FuelUsed"` and `"JumpDist"` properties. -* journal/Location: Strip out `"Wanted"`, `"Latitude"` and `"Longitude"` - properties. -* journal/Location and journal/FSDJump: strip out `"HappiestSystem"` - , `"HomeSystem"`, `"MyReputation"` and `"SquadronFaction"` properties within - the list of `"Factions"`. +The individual schemas will instruct you on various elisions (removals) to +be made to comply with this. Some of these requirements are also enforced by the schemas, and some things the schemas enforce might not be explicitly called out here, so **do** check what you're sending against the schema when implementing sending new events. + +## Receiving messages + +EDDN provides a continuous stream of information from uploaders. To use this +data you'll need to connect to the stream using ZeroMQ (a library is probably +available for your language of choice). + +The URL for the live Relay is: + + tcp://eddn.edcd.io:9500 + +Once you've connected to that you will receive messages. To access the +data you will first need to zlib-decompress each message. Then you will +have a textual JSON object as per the schemas. + +In general, check the guidance for [Uploading messages](#uploading-messages) +for the expected format of the messages. + +Consumers can utilise the `$schemaRef` value to determine which schema a +particular message is for. There is no need to validate the messages +against the schemas yourself, as that is performed on the EDDN Gateway. +Messages that do not pass the schema validation there are not forwarded to +receivers. + +There is [example code](https://github.com/EDCD/EDDN/tree/master/examples) +available for a variety of programming languages to help you get started. \ No newline at end of file diff --git a/schemas/blackmarket-README.md b/schemas/blackmarket-README.md new file mode 100644 index 0000000..0054bd1 --- /dev/null +++ b/schemas/blackmarket-README.md @@ -0,0 +1,55 @@ +# EDDN BlackMarket Schema + +## Introduction + +**This schema is deprecated. The same data has been made available via the +`prohibited` array in the [commodity schema](./commodity-v3.0.json) since +September 2017.** + +What follows below is only for historical curiosity. + +--- + +Here we document how to take data from an ED `MarketSell` Journal Event and +properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event `MarketSell`. + +### Key Renames +#### name +Due to how the EDDN schema is defined the `Type` key/value should +have the key renamed to `name`. + +#### prohibited +Due to how the EDDN schema is defined the `IllegalGoods` key/value should +have the key renamed to `prohibited`. + +#### marketID +The Journal documentation says this is `MarketID`, but in the schema the +`m` is lower case. + +### Elisions +You MUST remove the following key/value pairs from the data: + + - `TotalSale` + - `AvgPricePaid` + - `StolenGoods` + - `BlackMarket` - Because we're using this schema, so this is un-necessary. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### systemName +The star system name for where this market is. Use the `StarSystem` value +from the prior `Docked` or `Location` event. + +#### stationName +From the `StationName` value on the prior `Docked` or `Location` event. \ No newline at end of file diff --git a/schemas/codexentry-README.md b/schemas/codexentry-README.md index 69b6e4a..1f3725e 100644 --- a/schemas/codexentry-README.md +++ b/schemas/codexentry-README.md @@ -17,6 +17,12 @@ You MUST remove any key where the key name ends in You MUST remove the two keys `IsNewEntry` and `NewTraitsDiscovered`. ### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + #### StarPos You MUST **add** a `StarPos` key with value of type `array` containing the galaxy co-ordinates of the system. You will need to have obtained these diff --git a/schemas/commodity-README.md b/schemas/commodity-README.md new file mode 100644 index 0000000..c2a4516 --- /dev/null +++ b/schemas/commodity-README.md @@ -0,0 +1,94 @@ +# EDDN Commodity Schema + +## Introduction +Here we document how to take data from an ED `Market` Journal Event and +properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event `Market`, +and the additional file, `Market.json`, that it signals the writing of. + +So, look for the `Market` event, and when it occurs open and read the +`Market.json` file which contains the actual data. Treat *that* data as +the event. + +It *is* also possible to construct this data from a CAPI `/market` query. +See [Using CAPI data](#using-capi-data) below. + +### Statusflags +`statusFlags` is an optional augmentation that can be used to identify +`Producer`, `Consumer`, and `Rare` goods from the output of `Market.json`. +So, as per the schema, do include it if available. + +### Key Renames +Many of the key names have a different case defined in this schema, make +sure you are renaming them as appropriate. + +#### StarSystem to systemName +Rename the `StarSystem` key name to `systemName`. + +### Elisions +#### Remove _Localised key/values +All keys whose name ends with `_Localised`, i.e. the `Name_Localised` +key/values in Items. + +#### Other Elisions +You MUST remove the following key/value pairs from the data: + +- `StationType` key/value. +- `Producer` key/value pair in Items. +- `Rare` key/value pair in Items. +- `id` key/value pair in Items. + +In the list of commodites: + +- Skip commodities with `"categoryname": "NonMarketable"` (i.e. + Limpets - not purchasable in station market) or a *non-empty*`"legality":` + string (not normally traded at this station market). + +#### Item Category +Remove not only the `Category_Localised` key/value, as above, but also the +`Category` key/value pair from each Item. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +### Using CAPI data +It is *not* recommended to use CAPI data as the source as it's fraught with +additional issues. EDMarketConnector does so in order to facilitate +obtaining data without the player needing to open the commodities screen. + +Please read +[the guidance on checking for CAPI lag](README-EDDN-schemas.md#detecting-capi-data-lag) +before utilising CAPI data for EDDN messages. + +Note that CAPI `/market` data will sometimes have the `StatusFlasg` per +item, which are defined as optional in this schema (because they're not in +the Market.json data). You SHOULD include this data in your message if +using CAPI as the source. + +Now you will need to construct the necessary additional fields: + +#### CAPI horizons flag +If your application can be certain that the game client is still running, +and logged into the game (not just run to the main menu), then you can +simply use the value from the `LoadGame` journal event. + +Otherwise, you MUST check if any of the economies from the `/market` +data have a `name` of `Colony`, if so, set this flag true. + +Additionally, you should retrieve the CAPI `/shipyard` endpoint and check if +any of the listed ships or modules have a `sku` value of +`ELITE_HORIZONS_V_PLANETARY_LANDINGS`. If so, set this flag true. + +#### CAPI odyssey flag +Unfortunately there is no method to be *certain* of this from CAPI data, so +you will have to trust in the system/station name check and use the value +from the Journal `LoadGame` event. \ No newline at end of file diff --git a/schemas/fssdiscoveryscan-README.md b/schemas/fssdiscoveryscan-README.md new file mode 100644 index 0000000..6d4d844 --- /dev/null +++ b/schemas/fssdiscoveryscan-README.md @@ -0,0 +1,32 @@ +# EDDN FSSDiscoveryScan Schema + +## Introduction +Here we document how to take data from an ED `FSSDiscoveryScan` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event +`FSSDiscoveryScan`. + +### Key Renames +Many of the key names have a different case defined in this schema, make +sure you are renaming them as appropriate. + +### Elisions +You MUST remove the following key/value pairs from the data: + + - `Progress` key/value pair. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### StarPos +You MUST add a `StarPos` array containing the system co-ordinates from the +last `FSDJump`, `CarrierJump`, or `Location` event. \ No newline at end of file diff --git a/schemas/journal-README.md b/schemas/journal-README.md new file mode 100644 index 0000000..5dd7cea --- /dev/null +++ b/schemas/journal-README.md @@ -0,0 +1,86 @@ +# EDDN Journal Schema + +## Introduction +Here we document how to take data from miscellaneous ED Journal +events and properly structure it for sending to EDDN. + +This is the historical "all Journal events" schema that will be deprecated +in the future. Please check for a schema specific to the journal event +under consideration to see if data should be sent on that event specific +schema instead. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal events: + + - `Docked` + - `FSDJump` + - `Scan` + - `Location` + - `SAASignalsFound` + - `CarrierJump` + - `CodexEntry` - But see the separate + [codexentry schema](./codexentry-README.md) documentation. + +### Key Renames +Many of the key names have a different case defined in this schema, make +sure you are renaming them as appropriate. + +### Elisions +#### Remove _Localised key/values +All keys whose name ends with `_Localised`, i.e. the `Name_Localised` +key/values in Items. + +#### Personal data in `Docked` events +The following keys+values should be removed from `Docked` event data: + + - `Wanted` + - `ActiveFine` + - `CockpitBreach` + +#### Personal data in `FSDJump` events +The following keys+values should be removed from `FSDJump` event data: + +- `Wanted` +- `BoostUsed` +- `FuelLevel` +- `FuelUsed` +- `JumpDist` +- `HappiestSystem` from within the list of `Factions`. +- `HomeSystem` from within the list of `Factions`. +- `MyReputation` from within the list of `Factions`. +- `SquadronFaction` from within the list of `Factions`. + +#### Personal data in `Location` events +The following keys+values should be removed from `Location` event data: + +- `Wanted` +- `Latitude` +- `Longitude` +- `HappiestSystem` from within the list of `Factions`. +- `HomeSystem` from within the list of `Factions`. +- `MyReputation` from within the list of `Factions`. +- `SquadronFaction` from within the list of `Factions`. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### StarSystem +You MUST add a `StarSystem` key/value pair representing the name of the +system this event occurred in. Source this from either `Location`, +`FSDJump` or `CarrierJump` as appropriate. + +#### SystemAddress +You MUST add a `SystemAddress` key/value pair representing the numerical ID +of the system this event occurred in. Source this from either `Location`, +`FSDJump` or `CarrierJump` as appropriate. + +#### StarPos +You MUST add a `StarPos` array containing the system co-ordinates from the +last `FSDJump`, `CarrierJump`, or `Location` event. diff --git a/schemas/navbeaconscan-README.md b/schemas/navbeaconscan-README.md new file mode 100644 index 0000000..c12e91b --- /dev/null +++ b/schemas/navbeaconscan-README.md @@ -0,0 +1,31 @@ +# EDDN NavBeaconScan Schema + +## Introduction +Here we document how to take data from an ED `NavBeaconScan` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event +`NavBeaconScan`. + +### Elisions +There are no elisions in this schema. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### StarSystem +You MUST add a `StarSystem` key/value pair representing the name of the +system this event occurred in. Source this from either `Location`, +`FSDJump` or `CarrierJump` as appropriate. + +#### StarPos +You MUST add a `StarPos` array containing the system co-ordinates from the +last `FSDJump`, `CarrierJump`, or `Location` event. diff --git a/schemas/navroute-README.md b/schemas/navroute-README.md new file mode 100644 index 0000000..3025baa --- /dev/null +++ b/schemas/navroute-README.md @@ -0,0 +1,29 @@ +# EDDN NavRoute Schema + +## Introduction +Here we document how to take data from an ED `NavRoute` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the `NavRoute.json` file. That +it has been freshly written is signalled by the ED Journal event `NavRoute`. + +So, monitor the Journal as normal, and when you see a `NavRoute` event open +the `NavRoute.json` file for reading, read it, and close it again. Use the +data you got from reading this file, not merely the Journal event. + +The primary data to be sent is the `Route` array from the contents of the +separate file. + +### Elisions +There are no elisions in this schema. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. \ No newline at end of file diff --git a/schemas/outfitting-README.md b/schemas/outfitting-README.md new file mode 100644 index 0000000..090fc61 --- /dev/null +++ b/schemas/outfitting-README.md @@ -0,0 +1,42 @@ +# EDDN Outfitting Schema + +## Introduction +Here we document how to take data from an ED `Outfitting` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event +`Outfitting`. + +You MAY also source this data from the CAPI `/shipyard` endpoint. +Please read +[the guidance on checking for CAPI lag](README-EDDN-schemas.md#detecting-capi-data-lag) +before utilising CAPI data for EDDN messages. + +You only need the `name` key's value for each member of the `modules` array. + +### Key Renames +Many of the key names have a different case defined in this schema, make +sure you are renaming them as appropriate. + +### Elisions +Remove items whose availability depends on the Cmdr's status rather than on the +station. Namely: + +- Items that aren't weapons/utilities (`Hpt_*`), standard/internal + modules (`Int_*`) or armour (`*_Armour_*`) (i.e. bobbleheads, decals, + paintjobs and shipkits). This is enforced by the schema. +- Items that have a non-null `"sku"` property, unless + it's `"ELITE_HORIZONS_V_PLANETARY_LANDINGS"` (i.e. PowerPlay and tech + broker items). +- The `"Int_PlanetApproachSuite"` module (for historical reasons). + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. \ No newline at end of file diff --git a/schemas/scanbarycentre-README.md b/schemas/scanbarycentre-README.md new file mode 100644 index 0000000..da0e178 --- /dev/null +++ b/schemas/scanbarycentre-README.md @@ -0,0 +1,26 @@ +# EDDN ScanBaryCentre Schema + +## Introduction +Here we document how to take data from an ED `ScanBaryCentre` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event +`ScanBaryCentre`. + +### Elisions +There are no elisions in this schema. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### StarPos +You MUST add a `StarPos` array containing the system co-ordinates from the +last `FSDJump`, `CarrierJump`, or `Location` event. diff --git a/schemas/shipyard-README.md b/schemas/shipyard-README.md new file mode 100644 index 0000000..33016a4 --- /dev/null +++ b/schemas/shipyard-README.md @@ -0,0 +1,40 @@ +# EDDN Shipyard Schema + +## Introduction +Here we document how to take data from an ED `Shipyard` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event +`Shipyard`. + +You MAY also source this data from the CAPI `/shipyard` endpoint. +Please read +[the guidance on checking for CAPI lag](README-EDDN-schemas.md#detecting-capi-data-lag) +before utilising CAPI data for EDDN messages. + + You only need the `name` key's value for each member of the `PriceList` +array (if using Journal, it will be from the `ships` array if using CAPI +data). + +When using CAPI data *include* ships listed in the `"unavailable_list"` +property (i.e. available at this station, but not to this Cmdr). + +This list of ship names will go in the `ships` array in the EDDN message. + +### Key Renames +Many of the key names have a different case defined in this schema, make +sure you are renaming them as appropriate. + +### Elisions +There are no elisions in this schema. + +### Augmentations +#### horizons flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event. + +#### odyssey flag +You SHOULD add this key/value pair, using the value from the `LoadGame` event.