mirror of
https://github.com/andwati/gitea-mirror-sync.git
synced 2026-10-18 03:12:53 +00:00
Trim credentials before use; fix redact on awkward tokens
A GITEA_TOKEN secret stored with a trailing newline makes curl emit `Authorization: token <value>\n`. The bare LF terminates the header block early, so Content-Type and everything after it is parsed as request body and Gitea rejects the call with 422 "Empty Content-Type". GETs were unaffected because nothing they need follows the auth header, which is why this presented as a proxy stripping the header. - trim CR/LF and surrounding whitespace off GITHUB_TOKEN, GITEA_TOKEN, GITEA_URL and GITEA_OWNER, warning when a value needed it - rebuild redact() on bash substitution; the previous sed s/// blew up with "unterminated `s' command" on a token containing a newline - add AGENTS.md / CLAUDE.md documenting the run flow and these traps
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
# AGENTS.md
|
||||
|
||||
Guidance for coding agents working in this repository.
|
||||
|
||||
## What this is
|
||||
|
||||
One bash script, `mirror-github-to-gitea.sh`, that makes every GitHub repo owned by the
|
||||
token's user exist as a **pull mirror** on a Gitea instance, plus the GitHub Actions
|
||||
workflow (`.github/workflows/mirror-to-gitea.yml`) that runs it daily at 00:00 UTC.
|
||||
There is no build, no test suite, and no package manager. Dependencies are `bash`,
|
||||
`curl`, and `jq`.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# syntax check (the only "build" step)
|
||||
bash -n mirror-github-to-gitea.sh
|
||||
shellcheck mirror-github-to-gitea.sh # if installed
|
||||
|
||||
# full run — needs all four env vars
|
||||
GITHUB_TOKEN=... GITEA_URL=https://gitea.example.com GITEA_TOKEN=... GITEA_OWNER=... \
|
||||
./mirror-github-to-gitea.sh
|
||||
|
||||
# smoke-test a single repo instead of all of them (do this before any full run)
|
||||
ONLY=blog ./mirror-github-to-gitea.sh
|
||||
|
||||
# dump request/response headers for every Gitea write, tokens redacted
|
||||
DEBUG_HTTP=1 ONLY=blog ./mirror-github-to-gitea.sh
|
||||
|
||||
# trigger the workflow, optionally with header debugging on
|
||||
gh workflow run mirror-to-gitea.yml -f debug_http=true
|
||||
gh run watch
|
||||
```
|
||||
|
||||
## Environment contract
|
||||
|
||||
| Var | Source in CI | Notes |
|
||||
| --- | --- | --- |
|
||||
| `GITHUB_TOKEN` | `secrets.GH_PAT` | PAT with read access incl. private repos; also embedded in the migrate payload so Gitea can pull |
|
||||
| `GITEA_URL` | `secrets.GITEA_URL` | trailing slash is stripped — `//api/v1/...` gets mangled by some proxies |
|
||||
| `GITEA_TOKEN` | `secrets.GITEA_TOKEN` | Gitea token, repo scope |
|
||||
| `GITEA_OWNER` | `secrets.GITEA_OWNER` | defaults to `andwati` |
|
||||
| `MIRROR_INTERVAL` | — | Gitea sync interval, default `8h0m0s` |
|
||||
| `ONLY`, `DEBUG_HTTP` | `inputs.debug_http` | testing knobs |
|
||||
|
||||
`GITHUB_USER` is hardcoded in the script while the Gitea side comes from env, so the
|
||||
two accounts are configured differently — don't assume they're the same value.
|
||||
|
||||
## How a run works
|
||||
|
||||
1. Page through `GET /user/repos?affiliation=owner` (not `/users/{user}/repos`, which
|
||||
is public-only) and accumulate the list with `jq -s`.
|
||||
2. Per repo, `GET /api/v1/repos/{owner}/{name}` on Gitea to decide what to do:
|
||||
- **200** → compare `private` on both sides; `PATCH` the Gitea repo if they differ,
|
||||
otherwise skip.
|
||||
- **anything else** → treated as "not mirrored yet", so `POST /api/v1/repos/migrate`
|
||||
with `mirror: true`. A transient 5xx therefore looks like "missing" and produces a
|
||||
failed create rather than a retry.
|
||||
|
||||
The script only creates and adjusts visibility. It never deletes, renames, or unmirrors,
|
||||
and repos that exist on Gitea but not GitHub are left alone.
|
||||
|
||||
## Gotchas worth knowing before editing
|
||||
|
||||
- **Credentials are trimmed at startup, and that is load-bearing.** A GitHub secret
|
||||
saved with a trailing newline makes curl write `Authorization: token <value>\n`; the
|
||||
bare LF ends the header block early, so every header after it — `Content-Type`
|
||||
included — is parsed as body and Gitea answers `422 "[]: Empty Content-Type"`. `GET`s
|
||||
are unaffected because nothing they rely on follows the auth header, which is why this
|
||||
looked like a proxy problem at first. The script warns and trims; the real fix is
|
||||
re-saving the secret without the newline.
|
||||
- **`redact()` uses bash `${var//old/new}`, not `sed`.** Interpolating a secret into a
|
||||
`s///` expression breaks the moment the secret contains a slash or a newline.
|
||||
- **All body-bearing Gitea calls go through `gitea_write`.** It exists because Gitea
|
||||
answers `422 "[]: Empty Content-Type"` when a reverse proxy drops the header in front
|
||||
of it — which happened from GitHub runners but not from a local network. The helper
|
||||
sends the body with `--data-binary @file`, disables `Expect: 100-continue`, sets an
|
||||
explicit charset, and retries once over `--http1.1` when that error comes back. Don't
|
||||
replace it with an inline `curl -d`.
|
||||
- **`gitea_write` echoes the HTTP status on stdout and callers capture it with `$(...)`.**
|
||||
Any progress or diagnostic output added inside it must be redirected to `>&2`, or it
|
||||
gets swallowed into the status variable.
|
||||
- **The migrate payload contains the PAT.** Response bodies and header traces are piped
|
||||
through `redact()` before printing; keep any new output on that path. Bodies live in a
|
||||
`mktemp -d` cleaned by a trap, and are passed to curl by file rather than on the
|
||||
command line so tokens stay out of the process list.
|
||||
- The workflow's preflight step POSTs a harmless body to `/api/v1/markdown` over both
|
||||
HTTP/2 and HTTP/1.1. Reading its two status codes is the fastest way to tell a broken
|
||||
network path from a broken payload.
|
||||
- The script relies on `mirror-github-to-gitea.sh` being committed mode `100755`; the
|
||||
workflow invokes it directly with no `chmod`.
|
||||
|
||||
## Commits and PRs
|
||||
|
||||
Do not add self-attribution. No `Co-Authored-By: Claude ...` trailer on commits, and no
|
||||
"Generated with Claude Code" line in PR bodies — the message ends at its last
|
||||
substantive line. If a trailer slips through, amend and force-push with
|
||||
`--force-with-lease`.
|
||||
@@ -0,0 +1,7 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
See **[AGENTS.md](AGENTS.md)** for the commands, environment contract, run flow, and
|
||||
editing gotchas. That file is the single source of truth for this repo and applies here
|
||||
in full.
|
||||
@@ -15,11 +15,40 @@ ONLY="${ONLY:-}"
|
||||
# DEBUG_HTTP=1 prints the request/response headers of every write call (tokens redacted).
|
||||
DEBUG_HTTP="${DEBUG_HTTP:-}"
|
||||
|
||||
# --- Sanitise the credentials ---------------------------------------------
|
||||
# A secret pasted into GitHub Actions with a trailing newline is invisible in
|
||||
# the logs but fatal here: curl emits `Authorization: token <value>\n` and the
|
||||
# bare LF terminates the header block early, so every header curl writes after
|
||||
# it - Content-Type included - lands in the request body instead. Gitea then
|
||||
# rejects the call with 422 "Empty Content-Type". GETs survive because nothing
|
||||
# they need comes after the Authorization header.
|
||||
trim() { printf '%s' "$1" | tr -d '\r\n' | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'; }
|
||||
|
||||
for _var in GITHUB_TOKEN GITEA_TOKEN GITEA_URL GITEA_OWNER; do
|
||||
_raw="${!_var}"
|
||||
_clean=$(trim "$_raw")
|
||||
if [ "$_raw" != "$_clean" ]; then
|
||||
echo "Warning: ${_var} had surrounding whitespace or a newline; trimmed it." >&2
|
||||
echo " Fix the secret itself - re-add it without a trailing newline." >&2
|
||||
fi
|
||||
printf -v "$_var" '%s' "$_clean"
|
||||
done
|
||||
unset _var _raw _clean
|
||||
|
||||
GITEA_URL="${GITEA_URL%/}" # re-strip: trimming may have exposed a trailing slash
|
||||
|
||||
work_dir=$(mktemp -d)
|
||||
trap 'rm -rf "$work_dir"' EXIT
|
||||
|
||||
# Literal, delimiter-free substitution - building a sed s/// out of a secret
|
||||
# breaks as soon as the secret contains a slash or a newline.
|
||||
redact() {
|
||||
sed -e "s/${GITEA_TOKEN}/<GITEA_TOKEN>/g" -e "s/${GITHUB_TOKEN}/<GITHUB_TOKEN>/g"
|
||||
local line
|
||||
while IFS= read -r line || [ -n "$line" ]; do
|
||||
line=${line//"$GITEA_TOKEN"/<GITEA_TOKEN>}
|
||||
line=${line//"$GITHUB_TOKEN"/<GITHUB_TOKEN>}
|
||||
printf '%s\n' "$line"
|
||||
done
|
||||
}
|
||||
|
||||
# Send a JSON body to Gitea and echo the HTTP status.
|
||||
|
||||
Reference in New Issue
Block a user