mirror of
https://github.com/andwati/gitea-mirror-sync.git
synced 2026-10-18 03:12:53 +00:00
Trim credentials before use; fix redact on awkward tokens
A GITEA_TOKEN secret stored with a trailing newline makes curl emit `Authorization: token <value>\n`. The bare LF terminates the header block early, so Content-Type and everything after it is parsed as request body and Gitea rejects the call with 422 "Empty Content-Type". GETs were unaffected because nothing they need follows the auth header, which is why this presented as a proxy stripping the header. - trim CR/LF and surrounding whitespace off GITHUB_TOKEN, GITEA_TOKEN, GITEA_URL and GITEA_OWNER, warning when a value needed it - rebuild redact() on bash substitution; the previous sed s/// blew up with "unterminated `s' command" on a token containing a newline - add AGENTS.md / CLAUDE.md documenting the run flow and these traps
This commit is contained in:
@@ -0,0 +1,98 @@
|
|||||||
|
# AGENTS.md
|
||||||
|
|
||||||
|
Guidance for coding agents working in this repository.
|
||||||
|
|
||||||
|
## What this is
|
||||||
|
|
||||||
|
One bash script, `mirror-github-to-gitea.sh`, that makes every GitHub repo owned by the
|
||||||
|
token's user exist as a **pull mirror** on a Gitea instance, plus the GitHub Actions
|
||||||
|
workflow (`.github/workflows/mirror-to-gitea.yml`) that runs it daily at 00:00 UTC.
|
||||||
|
There is no build, no test suite, and no package manager. Dependencies are `bash`,
|
||||||
|
`curl`, and `jq`.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# syntax check (the only "build" step)
|
||||||
|
bash -n mirror-github-to-gitea.sh
|
||||||
|
shellcheck mirror-github-to-gitea.sh # if installed
|
||||||
|
|
||||||
|
# full run — needs all four env vars
|
||||||
|
GITHUB_TOKEN=... GITEA_URL=https://gitea.example.com GITEA_TOKEN=... GITEA_OWNER=... \
|
||||||
|
./mirror-github-to-gitea.sh
|
||||||
|
|
||||||
|
# smoke-test a single repo instead of all of them (do this before any full run)
|
||||||
|
ONLY=blog ./mirror-github-to-gitea.sh
|
||||||
|
|
||||||
|
# dump request/response headers for every Gitea write, tokens redacted
|
||||||
|
DEBUG_HTTP=1 ONLY=blog ./mirror-github-to-gitea.sh
|
||||||
|
|
||||||
|
# trigger the workflow, optionally with header debugging on
|
||||||
|
gh workflow run mirror-to-gitea.yml -f debug_http=true
|
||||||
|
gh run watch
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment contract
|
||||||
|
|
||||||
|
| Var | Source in CI | Notes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `GITHUB_TOKEN` | `secrets.GH_PAT` | PAT with read access incl. private repos; also embedded in the migrate payload so Gitea can pull |
|
||||||
|
| `GITEA_URL` | `secrets.GITEA_URL` | trailing slash is stripped — `//api/v1/...` gets mangled by some proxies |
|
||||||
|
| `GITEA_TOKEN` | `secrets.GITEA_TOKEN` | Gitea token, repo scope |
|
||||||
|
| `GITEA_OWNER` | `secrets.GITEA_OWNER` | defaults to `andwati` |
|
||||||
|
| `MIRROR_INTERVAL` | — | Gitea sync interval, default `8h0m0s` |
|
||||||
|
| `ONLY`, `DEBUG_HTTP` | `inputs.debug_http` | testing knobs |
|
||||||
|
|
||||||
|
`GITHUB_USER` is hardcoded in the script while the Gitea side comes from env, so the
|
||||||
|
two accounts are configured differently — don't assume they're the same value.
|
||||||
|
|
||||||
|
## How a run works
|
||||||
|
|
||||||
|
1. Page through `GET /user/repos?affiliation=owner` (not `/users/{user}/repos`, which
|
||||||
|
is public-only) and accumulate the list with `jq -s`.
|
||||||
|
2. Per repo, `GET /api/v1/repos/{owner}/{name}` on Gitea to decide what to do:
|
||||||
|
- **200** → compare `private` on both sides; `PATCH` the Gitea repo if they differ,
|
||||||
|
otherwise skip.
|
||||||
|
- **anything else** → treated as "not mirrored yet", so `POST /api/v1/repos/migrate`
|
||||||
|
with `mirror: true`. A transient 5xx therefore looks like "missing" and produces a
|
||||||
|
failed create rather than a retry.
|
||||||
|
|
||||||
|
The script only creates and adjusts visibility. It never deletes, renames, or unmirrors,
|
||||||
|
and repos that exist on Gitea but not GitHub are left alone.
|
||||||
|
|
||||||
|
## Gotchas worth knowing before editing
|
||||||
|
|
||||||
|
- **Credentials are trimmed at startup, and that is load-bearing.** A GitHub secret
|
||||||
|
saved with a trailing newline makes curl write `Authorization: token <value>\n`; the
|
||||||
|
bare LF ends the header block early, so every header after it — `Content-Type`
|
||||||
|
included — is parsed as body and Gitea answers `422 "[]: Empty Content-Type"`. `GET`s
|
||||||
|
are unaffected because nothing they rely on follows the auth header, which is why this
|
||||||
|
looked like a proxy problem at first. The script warns and trims; the real fix is
|
||||||
|
re-saving the secret without the newline.
|
||||||
|
- **`redact()` uses bash `${var//old/new}`, not `sed`.** Interpolating a secret into a
|
||||||
|
`s///` expression breaks the moment the secret contains a slash or a newline.
|
||||||
|
- **All body-bearing Gitea calls go through `gitea_write`.** It exists because Gitea
|
||||||
|
answers `422 "[]: Empty Content-Type"` when a reverse proxy drops the header in front
|
||||||
|
of it — which happened from GitHub runners but not from a local network. The helper
|
||||||
|
sends the body with `--data-binary @file`, disables `Expect: 100-continue`, sets an
|
||||||
|
explicit charset, and retries once over `--http1.1` when that error comes back. Don't
|
||||||
|
replace it with an inline `curl -d`.
|
||||||
|
- **`gitea_write` echoes the HTTP status on stdout and callers capture it with `$(...)`.**
|
||||||
|
Any progress or diagnostic output added inside it must be redirected to `>&2`, or it
|
||||||
|
gets swallowed into the status variable.
|
||||||
|
- **The migrate payload contains the PAT.** Response bodies and header traces are piped
|
||||||
|
through `redact()` before printing; keep any new output on that path. Bodies live in a
|
||||||
|
`mktemp -d` cleaned by a trap, and are passed to curl by file rather than on the
|
||||||
|
command line so tokens stay out of the process list.
|
||||||
|
- The workflow's preflight step POSTs a harmless body to `/api/v1/markdown` over both
|
||||||
|
HTTP/2 and HTTP/1.1. Reading its two status codes is the fastest way to tell a broken
|
||||||
|
network path from a broken payload.
|
||||||
|
- The script relies on `mirror-github-to-gitea.sh` being committed mode `100755`; the
|
||||||
|
workflow invokes it directly with no `chmod`.
|
||||||
|
|
||||||
|
## Commits and PRs
|
||||||
|
|
||||||
|
Do not add self-attribution. No `Co-Authored-By: Claude ...` trailer on commits, and no
|
||||||
|
"Generated with Claude Code" line in PR bodies — the message ends at its last
|
||||||
|
substantive line. If a trailer slips through, amend and force-push with
|
||||||
|
`--force-with-lease`.
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
# CLAUDE.md
|
||||||
|
|
||||||
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||||
|
|
||||||
|
See **[AGENTS.md](AGENTS.md)** for the commands, environment contract, run flow, and
|
||||||
|
editing gotchas. That file is the single source of truth for this repo and applies here
|
||||||
|
in full.
|
||||||
@@ -15,11 +15,40 @@ ONLY="${ONLY:-}"
|
|||||||
# DEBUG_HTTP=1 prints the request/response headers of every write call (tokens redacted).
|
# DEBUG_HTTP=1 prints the request/response headers of every write call (tokens redacted).
|
||||||
DEBUG_HTTP="${DEBUG_HTTP:-}"
|
DEBUG_HTTP="${DEBUG_HTTP:-}"
|
||||||
|
|
||||||
|
# --- Sanitise the credentials ---------------------------------------------
|
||||||
|
# A secret pasted into GitHub Actions with a trailing newline is invisible in
|
||||||
|
# the logs but fatal here: curl emits `Authorization: token <value>\n` and the
|
||||||
|
# bare LF terminates the header block early, so every header curl writes after
|
||||||
|
# it - Content-Type included - lands in the request body instead. Gitea then
|
||||||
|
# rejects the call with 422 "Empty Content-Type". GETs survive because nothing
|
||||||
|
# they need comes after the Authorization header.
|
||||||
|
trim() { printf '%s' "$1" | tr -d '\r\n' | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'; }
|
||||||
|
|
||||||
|
for _var in GITHUB_TOKEN GITEA_TOKEN GITEA_URL GITEA_OWNER; do
|
||||||
|
_raw="${!_var}"
|
||||||
|
_clean=$(trim "$_raw")
|
||||||
|
if [ "$_raw" != "$_clean" ]; then
|
||||||
|
echo "Warning: ${_var} had surrounding whitespace or a newline; trimmed it." >&2
|
||||||
|
echo " Fix the secret itself - re-add it without a trailing newline." >&2
|
||||||
|
fi
|
||||||
|
printf -v "$_var" '%s' "$_clean"
|
||||||
|
done
|
||||||
|
unset _var _raw _clean
|
||||||
|
|
||||||
|
GITEA_URL="${GITEA_URL%/}" # re-strip: trimming may have exposed a trailing slash
|
||||||
|
|
||||||
work_dir=$(mktemp -d)
|
work_dir=$(mktemp -d)
|
||||||
trap 'rm -rf "$work_dir"' EXIT
|
trap 'rm -rf "$work_dir"' EXIT
|
||||||
|
|
||||||
|
# Literal, delimiter-free substitution - building a sed s/// out of a secret
|
||||||
|
# breaks as soon as the secret contains a slash or a newline.
|
||||||
redact() {
|
redact() {
|
||||||
sed -e "s/${GITEA_TOKEN}/<GITEA_TOKEN>/g" -e "s/${GITHUB_TOKEN}/<GITHUB_TOKEN>/g"
|
local line
|
||||||
|
while IFS= read -r line || [ -n "$line" ]; do
|
||||||
|
line=${line//"$GITEA_TOKEN"/<GITEA_TOKEN>}
|
||||||
|
line=${line//"$GITHUB_TOKEN"/<GITHUB_TOKEN>}
|
||||||
|
printf '%s\n' "$line"
|
||||||
|
done
|
||||||
}
|
}
|
||||||
|
|
||||||
# Send a JSON body to Gitea and echo the HTTP status.
|
# Send a JSON body to Gitea and echo the HTTP status.
|
||||||
|
|||||||
Reference in New Issue
Block a user