This is a successor to #10805, which simply did not work. It is also much simpler and basically a one line change to enable an existing feature in [go-github](https://github.com/google/go-github).
Fixes#10845
With this fix and #10798 in place, a migration of a repo with ~3K issues and ~1.3k pull requests finally completed successfully.
## Patch
We use SleepUntilPrimaryRateLimitResetWhenRateLimited to instruct the go-github code to wait until the retry time and retry the request when the primary rate limit gets hit.
## Test case
TestGitHubDownloadRepo() has been modified such that 403 rate limit errors are injected every 7 requests with a retry time of one second, resulting in the rate limit condition being hit twice with the current tests. The test case confirms that the migration code itself is in fact unaffected by the rate limit being hit.
## Scope
This change does not affect secondary rate limits.
If the server is restarted during the wait for the rate limit refresh, the migration likely still fails when retried, because inserts for already present database objects will be attempted.
This approach effectively puts the task's goroutine to sleep until the retry time, which implies that the respective resources stay allocated.
A better approach might be to add the necessary infrastructure to support restarts of migration tasks at a later time, but this is much more involved, because the migration state would need to be saved and/or re-created based on already pulled data. This would also require adding support for database upserts.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/10846
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Nils Goroll <nils.goroll@uplex.de>
Co-committed-by: Nils Goroll <nils.goroll@uplex.de>
As mentioned in https://codeberg.org/forgejo/forgejo/issues/8131 and https://codeberg.org/forgejo/forgejo/issues/9018:
The github API changed and they now use cursor based pagination. So migration of issues could fail if there were about 10k resources to migrate.
What was done:
* Added a test for reproduction of the bug
* Updated the go-github library to v74
* Update api usage for Reactions
* Added a struct to GithubDownloaderV3 which holds cursorPagination related info
* Updated GetIssues to use cursorPagination
Caveats:
* So far, only listing issues supports the cursor method
* The test requires a valid access token to github as we need to access a repository with **a lot** of issues to test the issue
* We may want to skip this test in the pipeline
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9348
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: erik <erik_se@posteo.de>
Co-committed-by: erik <erik_se@posteo.de>
Change all license headers to comply with REUSE specification.
Fix#16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
A pr.Reviewer may be nil when migrating from Gitea if this is a team
request review.
We do not migrate teams therefore we cannot map these requests, but we can
migrate user requests.
Signed-off-by: Andrew Thornton <art27@cantab.net>
