Skip to content

EXP-01 Repository Sync

This experiment aims on discovering if its possible to have the aircraft GitHub repository (as well as all issues, and PRs) synced.

Plan

  • [COMPLETED - 19/08/2024] Import the https://github.com/flybywiresim/aircraft repository to GitLab.
  • [IN-PROGRESS] Create a sync-link that keeps the GitLab repository version up to date.
  • [PENDING] Test link by waiting for a commit, an issue and a pull request to be created in the GitHub repository, and observe they match the GitLab versions.

Repository Import

  • Import repository to GitLab [COMPLETED - 15/08/2024]
  • Confirm issues, PRs, commits, imported [IN PROGRESS] The importing went smoothly for the first few minutes, however it seemed to be stuck after a while and needed a page refresh.

import start

It seems also issues started to import after pull requests, but also notice how the amount of issues and pulls is much lower:

import after refresh

VS. the current amount of issues in the real repository:

gh repo issues

This is a significant difference that needs to be investigated.

Update: It seems GitLab has some throttledown to issues to prevent 429s from GitHub. After letting this run for a while, it seems perfectly healthy.

This behavior seems to be consisent with other sections (diffs) as shown below, so this is perfectly normal for this size of a repository I guess.

diff before

diff after

After quite some time, the process seemed frozen after a lot of tasks, and refreshing the page threw a 429 error.

Going to the dashboard though, I can now see the project, and when opening it I see the following:

import in progress repo

It seems there is still a lot of things to import, but so far issues and PRs look okay. It's evident such a big project as the aircraft repo takes A LOT of time to import. I think so far has been 4 hours.

About 40 minutes later, the host server had to be restarted. I am unsure how this has affected the import process, but it seems after GitLab was reboot it works fine.

The repository was imported succesfully, now it is out of sync as a few days have passed, however, the entire code, issues and PRs can all be seen! Major success so far!

Below as some performance metrics:

  • GitHub clone via CLI

github clone speed

Average speed: 23MiB/s (occasional dips to 12-13MiB/s every 6-7%) - Peaked at 29MiB/s Time elapsed: ~8 minutes

  • GitLab clone via CLI

gitlab clone speed

Average speed: 47MiB/s (occasional dips to 35MiB/s every 6-7%) - Peaked at 54MiB/s Time elapsed: ~4 minutes

Notes:

  1. The GitLab server seemed to run out of RAM during clones, with a more powerful machine this could probably be higher.
  2. My internet speed as of the tests (performed with connections to German servers, which was the origin connection from GitHub and GitLab server is on Germany..): ~95MiB/s
  3. Cloning was done on a fast machine with NVMe SSDs, no visible bottlenecks from CPU or RAM noticed.

Repository Linkage

Started 19-08-2024

As the import test is so far a success, it's time to figure out how I can keep the repository in-sync as right now there isn't any way to do this.

GitLab offers some methods for this, [Pull Linking] & [Push Linking], based on their documentation: https://docs.gitlab.com/ee/user/project/repository/mirror/index.html#authentication-methods-for-mirrors.

  • Option A: Push Linking - Can be used to mirror a repository from GitLab to another location.
  • Option B: Pull Linking - Can be used to mirror a repository from another location.
  • Option C: Bidirectional

For our usecase, we are going to be looking at the Pull Linking. However, as noted in their documentation, pull linking requires a premium or ultimate GitLab subscription. This is unfortunate, as it means we can not really use this feature out of the box, however, we can use the trial system for our experiments. Further discussion will need to be raised up for this.

For now, I am still working with FBW org admins to figure out what type of keys we will use. Though most probably I will use my personal account one which I also used for importing the repo initially.

Linkage Testing

TBD