From 028d478b0569598761c137ed753ceef8b3ad9218 Mon Sep 17 00:00:00 2001 From: andwati Date: Wed, 19 Aug 2026 08:44:58 +0300 Subject: [PATCH] Trim credentials before use; fix redact on awkward tokens A GITEA_TOKEN secret stored with a trailing newline makes curl emit `Authorization: token \n`. The bare LF terminates the header block early, so Content-Type and everything after it is parsed as request body and Gitea rejects the call with 422 "Empty Content-Type". GETs were unaffected because nothing they need follows the auth header, which is why this presented as a proxy stripping the header. - trim CR/LF and surrounding whitespace off GITHUB_TOKEN, GITEA_TOKEN, GITEA_URL and GITEA_OWNER, warning when a value needed it - rebuild redact() on bash substitution; the previous sed s/// blew up with "unterminated `s' command" on a token containing a newline - add AGENTS.md / CLAUDE.md documenting the run flow and these traps --- AGENTS.md | 98 +++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 7 +++ mirror-github-to-gitea.sh | 31 ++++++++++++- 3 files changed, 135 insertions(+), 1 deletion(-) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..fed24e8 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,98 @@ +# AGENTS.md + +Guidance for coding agents working in this repository. + +## What this is + +One bash script, `mirror-github-to-gitea.sh`, that makes every GitHub repo owned by the +token's user exist as a **pull mirror** on a Gitea instance, plus the GitHub Actions +workflow (`.github/workflows/mirror-to-gitea.yml`) that runs it daily at 00:00 UTC. +There is no build, no test suite, and no package manager. Dependencies are `bash`, +`curl`, and `jq`. + +## Commands + +```bash +# syntax check (the only "build" step) +bash -n mirror-github-to-gitea.sh +shellcheck mirror-github-to-gitea.sh # if installed + +# full run — needs all four env vars +GITHUB_TOKEN=... GITEA_URL=https://gitea.example.com GITEA_TOKEN=... GITEA_OWNER=... \ + ./mirror-github-to-gitea.sh + +# smoke-test a single repo instead of all of them (do this before any full run) +ONLY=blog ./mirror-github-to-gitea.sh + +# dump request/response headers for every Gitea write, tokens redacted +DEBUG_HTTP=1 ONLY=blog ./mirror-github-to-gitea.sh + +# trigger the workflow, optionally with header debugging on +gh workflow run mirror-to-gitea.yml -f debug_http=true +gh run watch +``` + +## Environment contract + +| Var | Source in CI | Notes | +| --- | --- | --- | +| `GITHUB_TOKEN` | `secrets.GH_PAT` | PAT with read access incl. private repos; also embedded in the migrate payload so Gitea can pull | +| `GITEA_URL` | `secrets.GITEA_URL` | trailing slash is stripped — `//api/v1/...` gets mangled by some proxies | +| `GITEA_TOKEN` | `secrets.GITEA_TOKEN` | Gitea token, repo scope | +| `GITEA_OWNER` | `secrets.GITEA_OWNER` | defaults to `andwati` | +| `MIRROR_INTERVAL` | — | Gitea sync interval, default `8h0m0s` | +| `ONLY`, `DEBUG_HTTP` | `inputs.debug_http` | testing knobs | + +`GITHUB_USER` is hardcoded in the script while the Gitea side comes from env, so the +two accounts are configured differently — don't assume they're the same value. + +## How a run works + +1. Page through `GET /user/repos?affiliation=owner` (not `/users/{user}/repos`, which + is public-only) and accumulate the list with `jq -s`. +2. Per repo, `GET /api/v1/repos/{owner}/{name}` on Gitea to decide what to do: + - **200** → compare `private` on both sides; `PATCH` the Gitea repo if they differ, + otherwise skip. + - **anything else** → treated as "not mirrored yet", so `POST /api/v1/repos/migrate` + with `mirror: true`. A transient 5xx therefore looks like "missing" and produces a + failed create rather than a retry. + +The script only creates and adjusts visibility. It never deletes, renames, or unmirrors, +and repos that exist on Gitea but not GitHub are left alone. + +## Gotchas worth knowing before editing + +- **Credentials are trimmed at startup, and that is load-bearing.** A GitHub secret + saved with a trailing newline makes curl write `Authorization: token \n`; the + bare LF ends the header block early, so every header after it — `Content-Type` + included — is parsed as body and Gitea answers `422 "[]: Empty Content-Type"`. `GET`s + are unaffected because nothing they rely on follows the auth header, which is why this + looked like a proxy problem at first. The script warns and trims; the real fix is + re-saving the secret without the newline. +- **`redact()` uses bash `${var//old/new}`, not `sed`.** Interpolating a secret into a + `s///` expression breaks the moment the secret contains a slash or a newline. +- **All body-bearing Gitea calls go through `gitea_write`.** It exists because Gitea + answers `422 "[]: Empty Content-Type"` when a reverse proxy drops the header in front + of it — which happened from GitHub runners but not from a local network. The helper + sends the body with `--data-binary @file`, disables `Expect: 100-continue`, sets an + explicit charset, and retries once over `--http1.1` when that error comes back. Don't + replace it with an inline `curl -d`. +- **`gitea_write` echoes the HTTP status on stdout and callers capture it with `$(...)`.** + Any progress or diagnostic output added inside it must be redirected to `>&2`, or it + gets swallowed into the status variable. +- **The migrate payload contains the PAT.** Response bodies and header traces are piped + through `redact()` before printing; keep any new output on that path. Bodies live in a + `mktemp -d` cleaned by a trap, and are passed to curl by file rather than on the + command line so tokens stay out of the process list. +- The workflow's preflight step POSTs a harmless body to `/api/v1/markdown` over both + HTTP/2 and HTTP/1.1. Reading its two status codes is the fastest way to tell a broken + network path from a broken payload. +- The script relies on `mirror-github-to-gitea.sh` being committed mode `100755`; the + workflow invokes it directly with no `chmod`. + +## Commits and PRs + +Do not add self-attribution. No `Co-Authored-By: Claude ...` trailer on commits, and no +"Generated with Claude Code" line in PR bodies — the message ends at its last +substantive line. If a trailer slips through, amend and force-push with +`--force-with-lease`. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..95be6e8 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,7 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +See **[AGENTS.md](AGENTS.md)** for the commands, environment contract, run flow, and +editing gotchas. That file is the single source of truth for this repo and applies here +in full. diff --git a/mirror-github-to-gitea.sh b/mirror-github-to-gitea.sh index bf6faae..42a3b8e 100755 --- a/mirror-github-to-gitea.sh +++ b/mirror-github-to-gitea.sh @@ -15,11 +15,40 @@ ONLY="${ONLY:-}" # DEBUG_HTTP=1 prints the request/response headers of every write call (tokens redacted). DEBUG_HTTP="${DEBUG_HTTP:-}" +# --- Sanitise the credentials --------------------------------------------- +# A secret pasted into GitHub Actions with a trailing newline is invisible in +# the logs but fatal here: curl emits `Authorization: token \n` and the +# bare LF terminates the header block early, so every header curl writes after +# it - Content-Type included - lands in the request body instead. Gitea then +# rejects the call with 422 "Empty Content-Type". GETs survive because nothing +# they need comes after the Authorization header. +trim() { printf '%s' "$1" | tr -d '\r\n' | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'; } + +for _var in GITHUB_TOKEN GITEA_TOKEN GITEA_URL GITEA_OWNER; do + _raw="${!_var}" + _clean=$(trim "$_raw") + if [ "$_raw" != "$_clean" ]; then + echo "Warning: ${_var} had surrounding whitespace or a newline; trimmed it." >&2 + echo " Fix the secret itself - re-add it without a trailing newline." >&2 + fi + printf -v "$_var" '%s' "$_clean" +done +unset _var _raw _clean + +GITEA_URL="${GITEA_URL%/}" # re-strip: trimming may have exposed a trailing slash + work_dir=$(mktemp -d) trap 'rm -rf "$work_dir"' EXIT +# Literal, delimiter-free substitution - building a sed s/// out of a secret +# breaks as soon as the secret contains a slash or a newline. redact() { - sed -e "s/${GITEA_TOKEN}//g" -e "s/${GITHUB_TOKEN}//g" + local line + while IFS= read -r line || [ -n "$line" ]; do + line=${line//"$GITEA_TOKEN"/} + line=${line//"$GITHUB_TOKEN"/} + printf '%s\n' "$line" + done } # Send a JSON body to Gitea and echo the HTTP status.