EDMarketConnector/docs/Releasing.md

# Introduction

Builds can, **and must in normal operation** be run automatically from GitHub 
Actions. For more information on that process
see [Automatic Builds](https://github.com/EDCD/EDMarketConnector/blob/main/docs/Automatic%20Builds.md).
This allows us to state that the files we distribute never touched anything 
but GitHub servers before a user downloaded them, which means no 
possibility of malware on a developer's machine infecting the resulting files.

Obviously you might still need to run a manual build on your own hardware 
in order to test changes and/or diagnose build issues.  As such
this document aims to enable anyone to quickly get up to speed on how to:

1. Build a Windows .exe for the application
1. Package that .exe into an .msi file for distribution
1. Handle the files generated so the application automatically detects new
 available versions and asks the user to upgrade.

Note that for Windows only a 32-bit application is supported at this time.
This is principally due to the Windows Registry handling in
`config/windows.py`.


# Environment

You will need several pieces of software installed, or the files from their
.zip archives, in order to build the .exe and generate the .msi

1. [WinSparkle](https://github.com/vslavik/winsparkle): `winsparkle.dll` and
 `winsparkle.pdb` from the release's .zip file.  v0.7.0 is the most recently
 tested version.  Copy the two files, found at `<zip file>\<version>\Release`,
 into your checkout of the EDMC git files.
1. [Windows SDK](https://developer.microsoft.com/en-US/windows/downloads/windows-10-sdk/).
 This is needed for the internationalisation support in EDMC.
 [Windows 10 SDK, version 2004 (10.0.19041.0)](https://go.microsoft.com/fwlink/p/?linkid=2120843)
 is the most recently tested version.  Technically you only need the following
 components: `MSI Tools`, `Windows SDK for Desktop C++ x86 Apps` (which will
 auto-select some others).  NB: If you have need to uninstall this it's
 "Windows Software Development Kit - Windows 10.0.19041.1" in
 "Apps & Features", *not* "Windows SDK AddOn".
1. [Python](https://python.org): 32-bit version of Python 3.10 for Windows.
 [v3.10.3](https://www.python.org/downloads/release/python-3103/) is the most
 recently tested version.  You need the `Windows x86 executable installer`
 file, for the 32-bit version.  Double-check the version against the
   `.python.version` file, as it should always contain the intended version.
1. [py2exe](https://github.com/albertosottile/py2exe) - Now available via PyPi,
 so will be picked up with the `pip install` below.  Latest tested as per
 `requirements-dev.txt`.

1. You'll now need to 'pip install' several python modules.
    1. Ensure you have `pip` installed. If needs be see
     [Installing pip](https://pip.pypa.io/en/stable/installing/)
    1. The easiest way is to utilise the `requirements-dev.txt` file:
     `python -m pip install --user -r requirements-dev.txt`. This will install
     all dependencies plus anything required for development.
    1. Else check the contents of both `requirements.txt` and `requirements-dev.txt`,
     and ensure the modules listed there are installed as per the version
     requirements.

If you are using different versions of any of these tools then please ensure
that the paths where they're installed match the associated lines in
`build.py`.  i.e. if you're using later Windows SDK kit you might need to edit
the SDKPATH line.

# Version Strings

This project now uses strict [Semantic Version](https://semver.org/#semantic-versioning-specification-semver)
version strings.

1. **Version strings should always be referred to as, e.g. `Major.Minor.Patch`
 not the old `A.BC` scheme, nor the pre-Semantic Version `A.B.C.D` scheme.**
1. Any stable release should have a version of **only** `Major.Minor.Patch`,
 correctly incrementing depending on the changes since the last stable release.
1. For any pre-release again increment the `Major.Minor.Patch` as fits the
 changes since the last *stable* release.
1. Any pre-release should have a <pre-release> component of either:
    1. `-beta<serial>`, i.e. `-beta1`.  This should be used when first asking
     a wider audience to test forthcoming changes.
    1. `-rc<serial>`, i.e. `-rc1`.  This is used when testing has shown this
     code should be ready for full release, but you want even wider testing.

    In both these cases simply increment `<serial>` for each new release.  *Do*
    start from `1` again when beginning `-rc` releases.


# Necessary Edits

There are some things that you should always change before running your own
version of EDMC

1. The Frontier CAPI client ID.  This is hardcoded in companion.py, but can be
 overridden by setting a CLIENT_ID environment variable.

There are other things that you should probably change, but can get away with
leaving at the upstream values, especially if you only you are going to use the
resulting .exe and/or .msi files. **But** realise that the resulting program
will still try to check for new versions at the main URL unless you change
that.

1. Company is set in  `build.py`. Search for `company_name`.  This
 is what appears in the EXE properties, and is also used as the location of
 WinSparkle registry entries on Windows.

1. Application names, version and URL of the file with latest release
 information. These are all in the `config/__init__.py` file.  See the
 `from config import ...` lines in `build.py`:
    1. `appname`: The short appname, e.g. 'EDMarketConnector'
    2. `applongname`: The long appname, e.g. 'E:D Market Connector'
    3. `appcmdname`: The CLI appname, e.g. 'EDMC'
    4. `_static_appversion`: The current version, e.g. `4.0.2`.  **You MUST
     make this something like `4.0.2+<myversion>` to differentiate it from
     upstream.**  Whatever is in this field is what will be reported if
     sending messages to EDDN, or any of the third-party website APIs.
     This is utilising the 'build metadata' part of a Semantic version.
    5. `copyright`: The Copyright string.
    6. `update_feed`: The URL where the application looks for current latest
     version information.  This URL should be hosting a renamed (so the full
     URL doesn't change over application versions) version of the
     appcast_win_<version>.xml file.  The original upstream value is
     `https://raw.githubusercontent.com/EDCD/EDMarketConnector/releases/edmarketconnector.xml`.

2. Location of release files.  This needs to be cited correctly in the
   `edmarketconnector.xml` file, which is what the application queries to
   see if there is a newer version.
   Look for the `url="...` line in the `<enclosure ...` that is like:

    ```xml
    <enclosure
        url="https://github.com/EDCD/EDMarketConnector/releases/download/Release/4.2.3/EDMarketConnector_win_4.2.3.msi"
        sparkle:os="windows"
        sparkle:installerArguments="/passive LAUNCH=yes"
        sparkle:version="4.2.3"
        length="11382784"
        type="application/octet-stream"
    />
    ```


## Adding a new file

If you add a new file to the program that needs to be distributed to users as
well then you will need to properly add it to the build process.

### build.py

You'll need to add it in `build.py` so that py2exe includes it in
the build.  Add the file to the DATA_FILES statement.

# Pre-Packaging Steps

Before you create a new install each time you should:

1. Ensure the data sourced from coriolis.io is up to date and works:
2. Update the `coriolis-data` repo. **NB: You will need 'npm' installed for
    this.**
   1. `cd coriolis-data`
   2. `git pull`
   3. `npm install` - to check it's worked.
3. Run `coriolis-update-files.py` to update `modules.json` and `ships.json`. **NB:
    The submodule might have been updated by a GitHub workflow/PR/merge, so
    be sure to perform this step for every build.**
4. XXX: Test ?
5. `git commit` the changes to the repo and the `.p` files.
6. Ensure translations are up to date, see [Translations.md](Translations.md).

# Preparing to Package

We'll use an old version string, `4.0.2`, as an example throughout the
following.

1. You should by this time know what changes are going into the release, and
which branch (stable or beta) you'll be ultimately updating.
2. So as to make backing out any mistakes easier create a new branch for this
release, using a name like `release-4.0.2`.  Do not use the tag
`Release/4.0.2` form, that could cause confusion.
    1. `git checkout stable` # Or whichever other branch is appropriate.
    1. `git pull origin` # Ensures local branch is up to date.
    1. `git checkout -b release-4.0.2`

3. Get all the relevant code changes into this branch.  This might mean
merging from another branch, such as an issue-specific one, or possibly
cherry-picking commits.  See [Contributing Guidelines](../Contributing.md)
for how such branches should be named.

4. You should have already decided on the new
[Version String](#Version-Strings), as it's specified in `config/__init__.py`.
You'll need to redo the `.msi` build if you forgot. **Remember to do a fresh
git commit for this change.**

5. Prepare a changelog text for the release.  You'll need this both for the
GitHub release and the contents of the `edmarketconnector.xml` file if making
a `stable` release, as well as any social media posts you make.
    1. The primary location of the changelog is [Changelog.md](../Changelog.md) -
    update this first.
    1. To be sure you include all the changes look at the git log since the
    prior appropriate (pre-)release.
    1. As you're working in a version-specific branch, `release-4.0.2`, you
    can safely commit these changes and push to GitHub.
     **Do not merge the branch with `releases` until the GitHub release is in place.**

If you're wondering, you needed to get the changelog prepared before building
the .exe and .msi because ChangeLog.md is bundled with the install.


# Adding killswitches 

If anything in this new release addresses a bug that causes, e.g. bad data
to be sent over EDDN, then you should add an appropriate entry to the
killswitches.json file *in the `releases` branch*.  That file **must only ever
be committed to the `releases` branch!!!**  See [docs/Killswitches.md](Killswitches.md).

Killswitch files can and should be verified using the `killswitch_test.py`
script in the `scripts` directory

# Packaging & Installer Generation

You'll want to do the .exe and .msi generation in a `cmd.exe` window, not e.g.
a 'Git bash' window.  The 'Terminal' tab of PyCharm works fine.

Assuming the correct python.exe is associated with .py files then simply run:

```batch
build.py
```

else you might need this, which assumes correct python.exe is in your PATH:

```batch
python.exe build.py
```

else you'll have to specify the path to python.exe, e.g.:

```batch
"C:\Program Files \(x86)\Python38-32\python.exe" build.py
```

Output will be something like (`...` denoting parts elided for brevity):

```plaintext
Git short hash: 993f946b.DIRTY
INFO:runtime:Analyzing the code
INFO:runtime:Found 695 modules, 60 are missing, 0 may be missing
...
Building 'dist.win32\EDMC.exe'.
Building 'dist.win32\EDMarketConnector.exe'.
```

**Do check the output** for things like not properly specifying extra files
to be included in the install.  If they're not picked up by current rules in
`build.py` then you will need to add them to the `win32`
`DATA_FILES` array.

You should now have one new/updated folder `dist.win32` and two new files
(version string dependent): `EDMarketConnector_win_4.0.2.msi` and
`appcast_win_4.0.2.xml`.

Check that the `EDMarketConnector.exe` in the `dist.win32` folder does run
without errors.

Finally, uninstall your current version of ED Market Connector and re-install
using the newly generated `EDMarketConnector_installer_4.0.2.exe` file.  Check the
resulting installation does work (the installer will run the program for you).

Update `edmarketconnector.xml` once more to set the `length=` attribute of the
enclosure to match the file size of the `EDMarketConnector_installer_4.0.2.exe` file.
The git commit for this should end up being the release tag as below.

# Distribution

Whether you built it manually or automatically you **MUST** test the `.exe` 
installer file prior to making the release live.

Once that is done then for manually built installers:

1. Add a git tag for the release, which you'll refer to when actually creating
    the release:
        1. This should be named `Release/A.B.C`, e.g. `Release/4.0.2.` as per
        the version string.

    **Do NOT add this tag until you're sure you're ready.  Pushing a tag to 
    GitHub that matches the pattern `Release/*` (double-check this in
    [the GitHub Windows Build Action file](../.github/workflows/windows-build.yml))
    will cause an auto-build and creation of a draft release.**

    Yes, this does mean you should really just be using this auto-build setup
    when creating an installer for release to users.  You'll at least need to
    edit the draft release that it creates:

      1. Swap out its `.msi` for the one that you built.
      2. Create a matching `hashes.sum` file for your `.msi` file:
   
             sha256sum EDMarketConnector_win*.msi > ./hashes.sum

          and replace the one in the draft release with this.

    But, **again, you should just be using the auto-build
    mechanism**.

3. Now push the release-specific branch to GitHub.
    1. Check which of your remotes is for github with `git remotes -v`. It
    should really be `origin` and the following assumes that.
    1. `git push --set-upstream --tags origin release-4.0.2`

4. Merge the release-specific branch into the appropriate `stable` or `beta`
branch.  You can either do this locally and push the changes, or do it on
GitHub.  You'll want to reference `stable` or `beta` in the next step, *not
the release-4.0.2 branch, as it's temporary.*

5. **You should no longer need to manually create a release, due to 
    auto-building of any release tag, but you'll probably still need to edit
    in the ChangeLog, so...**

    Craft a [new github Release](https://github.com/EDCD/EDMarketConnector/releases/new),
    1. Use the new tag so as to reference the correct commit, along with the
    appropriate `stable` or `beta` branch as the 'Target'.
    2. Use the changelog text you already prepared to fill in the 'Release
    title' and description.
    3. Attach the `EDMarketConnector_win_<version>.msi` file for Windows (the
    Source Code files are added by github based on the release tag).
    4. **If you are making a `beta` or otherwise pre-release you MUST tick the
    `[ ] This is a pre-release` box.**  Not doing so will cause this release
    to be pointed to by the 'latest' URL.
    5. We always create a discussion for any new release, so tick the
      `Create a discussion for this release` box, and check it's targeted at
      the `Announcement` category.

Once the release is created, then **only if making a `stable` release**
update `edmarketconnector.xml` **in the `releases` branch only** to add this
changelog text to the correct section(s):
1. `git checkout releases`
2. `git merge stable` - You should have merged the new release branch 
   into `stable` above.
3. Use the following to generate HTML from the MarkDown (`pip
  install grip` first if you need to):
    `grip --export ChangeLog.md`

4. Open `edmarketconnector.xml` in your editor.
5. If there's still a Mac OS section croll down past it to the Windows
  section.
6. You'll need to change the `<title>` and `<description>` texts to
  reflect the latest version and the additional changelog.
7. Update the `url` and `sparkle:version` elements of the `<enclosure>`
  section.
    1. The `url` needs to match what GitHub created in the Release for the
        `.msi` file. Check it!

        If, for instance, you fail to *update* this URL then upon running the 'new'
        installer it will silently fail, because you made people try to install
        the old version over the old version.
    2. Yes, `sparkle:version` should be the Semantic Version string,
       not the Windows A.B.C.D form.
8. `git push origin`

   This is the final step that fully publishes the release for running
   EDMC instances to pick up on 'Check for Updates'.  The WinSparkle check for
   updates specifically targets:

    `https://raw.githubusercontent.com/EDCD/EDMarketConnector/releases/edmarketconnector.xml`

   as per `config/__init__.py` `update_feed`.

   NB: It can take some time for GitHub to show the changed
   edmarketconnector.xml contents to all users.

**You should now update [Known Issues](https://github.com/EDCD/EDMarketConnector/issues/618)
to reflect anything now fixed in latest release.**

# Pre-Releases

If you are making a pre-release then:

1. **DO NOT** Edit `edmarketconnector.xml` at all.  No, not even if you 
  think you won't accidentally merge it into `releases`. Just don't change it
   at all.
1. **DO NOT** merge into `releases`.
1. **DO NOT** merge into `stable`.
1. *Do* merge the code into `beta` after you have made a 'pre-release' on
 GitHub.

# Changing Python version

When changing the Python version (Major.Minor.Patch) used:

1. Change the contents of `.python-version` so that pyenv notices.  All of
  the GitHub workflows now reference this via the `setup-python`
  `python-version-file` directive.

1. Any version change:

   1. `ChangeLog.md` - The `We now test against, and package with, Python
       M.m.P.` line.

1. Major or Minor level changes:

    1. `build.py` will need its version check updating.
    2. `.pre-commit-config.yaml` will need the `default_language_version`
       section updated to the appropriate version.