Repository navigation
Find the Maven artifacts a pinned rules_jvm_external hub builds in scan - #2191
Mikola Lysenko (mikolalysenko) wants to merge 4 commits into
Conversation
`socket-patch scan` takes its Maven packages from the JVM caches of a project that has a pom.xml, a Gradle build or a Scala-tool build. A checkout whose jars only a rules_jvm_external hub fetches has none of those, so a hosted scan there knows no package and never asks the API, although `get <purl> --mode hosted` in the same directory pins the same jar in MODULE.bazel. Two flows of tests/e2e_bazel_build/maven.rs now hold that on real Bazel, so that a change which makes scan read the hub's lock has expectations to turn over: - a_hosted_scan_of_a_tree_only_bazel_builds (new): after the ruleset's pin and a build, `scan --mode hosted --yes` exits 0 with scannedPackages 0, no package and no event, sends no batch request and changes no file, and the build prints the upstream marker as before. The stand-in's API holds a patch for the victim the whole time. - a_version_bump_of_the_patched_artifact: the build stops at the stale entry and `rollback <purl>` takes it out, as before. The flow then puts the stale entry back, as the file is for a user who has not run the rollback. The same scan finds no package there either and leaves the entry at which Bazel stops, although the API also grants a patch for the release the lock now pins (1.11.0). For that the stand-in grants two patches of the victim, one per release: grant() takes the version and hands the patch back, and the caller mounts the API once with both, because a second mount does not replace the answers of the first. Measured with Bazel 9.3.2 on macOS arm64, two flows at a time: the five Maven flows pass in 21.5, 24.3, 24.9, 30.9 and 22.1 seconds, and none of the 166 requests behind a logging proxy went to a public Maven repository (from a cold repository cache: 64.0, 71.6, 34.2, 60.5 and 29.8 seconds, 208 requests). No product code changes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`socket-patch scan` takes its Maven packages from the JVM caches of a
project that has a pom.xml, a Gradle build or a Scala-tool build. In a
tree that only Bazel builds it said "No packages found. Run your package
manager's install first.", although `get <purl> --mode hosted` in the
same directory pins that jar in MODULE.bazel: a user had to name every
purl. The lock of a pinned hub (maven_install.json) names each jar that
Bazel fetches, at the version it builds.
What changes:
- A hosted scan at the root of a Bazel workspace takes the jars that the
pinned hubs fetch for packages of the project, beside the ones it
finds installed (bazel::maven::hub_jars, through scan's lockfile
supplement). A hub's jar is the one without a classifier of an
artifact that the hub's lock has a sha256 for, does not skip, and
lists under a repository (or takes from the Maven local repository);
its version is the lock's. It counts in scannedPackages, and in
lockfileOnlyPackages with notInstalled unless a crawl found that
version installed. A patch found for it is pinned as `get` pins it:
one block in MODULE.bazel. A jar whose repository has no known name
is a package too: a patch found for it gets maven_bazel_artifact.
- The scan takes a stale entry out of the block. When the lock of a hub
moves on (a version bump, an artifact that was removed), Bazel stops
at the entry of the release it held ("does not generate repository").
A scan that only pinned the new release beside it would report
`applied` and leave a build that fails. So the scan restores such an
entry before the rewrite, through the sweep that restores a withdrawn
pin (scan/withdrawn.rs, whose messages for a withdrawn pin are the
ones they were): the entry alone, under the apply lock, previewed by
--dry-run, one rolledBack event with the reason
maven_bazel_entry_stale, also on a scan that finds no package. An
entry is stale only when every hub was read and none fetches its jar
at that version; a pin of the same patch in a pom.xml stays.
- `get <advisory> --mode hosted` keeps the releases that a pinned hub
fetches, by the same supplement.
- No jar is guessed for a hub that was not read with certainty (no
lock_file, a lock or a tag that cannot be read, a maven_install of a
WORKSPACE file). The run gives one warning, maven_bazel_hub_not_read,
with the file, the line and the reason, and still looks up the jars of
the hubs it read. With no package at all a person reads "No packages
found (see the warning)." in the place of the advice to install. The
warning is for a workspace that has to do with the ruleset: a call
names rules_jvm_external, the WORKSPACE file calls maven_install, or a
maven_install.json lies at the root. A Bazel workspace of other
rulesets with a file that cannot be read scans as it did.
- A bare scan covers the project directories below --cwd, and --cwd when
it is a project. The root of a Bazel workspace needs no lockfile or
build file of its own, so a bare hosted scan from a root above other
projects scanned those and never the root, while the same scan with
--json scanned the root. The root is now a project of a hosted scan
when its hubs have jars, something of them was not read, or the block
has a stale entry.
- Every hosted scan at a Bazel root now reads the hubs, so Hubs::read
reads each lock once for all the hubs that name it and at most 64 MiB
of locks for one workspace (the lock past that is a doubt), and the
scan walks each lock once. With a debug build, a checkout of a
hundred hubs on one 12.8 MiB lock of 80,000 artifacts scans in 1.5 s
with 175 MB, as one hub does (1.0 s), and twenty hubs with twenty
such locks in 1.7 s with 426 MB.
- No other run reads a hub for packages: a scan in vendored or agent
mode, one whose --ecosystems leaves Maven out, a --global one, one of
a project below the Bazel root, and every run of a project in no Bazel
workspace are the runs they were.
A hub's jar with a patch is a patch of the run like any other, so the
scan writes for it what `get <purl> --mode hosted` writes in that
checkout: beside a Gradle build the hosted Gradle pin too
(settings.gradle and three files under .socket/gradle), and beside a
pom.xml what the pom lane pins or says of that pom.
Not in this change: `get <purl>` leaves a stale entry where it is (a
hosted scan, rollback and remove take it out), hubs that other Bazel
modules declare or fill are not read, and the in-memory engine reads no
hub.
Tests:
- bazel::maven: one table of nineteen workspaces for what counts as a
jar of a hub (a lock over 1 MiB among them), what is said of a hub
that was not read and where nothing is said, and which entry is stale;
nothing for a project below the root or without a workspace. One test
for the locks: read once, and within the bound.
- tests/e2e_bazel_maven_beside.rs, nine tests without a Bazel: the scan
as a program and as a person, as a dry run and a second time; a root
with a lockfile of its own, alone and above another project; a stale
entry, with and without a patch of the new release, with no package
at all, while a hub is in doubt, and beside a pom that keeps its pin;
a Gradle build and a pom beside the hub, each compared with what `get
<purl>` writes and says there; a pom that selects another release;
each way a hub is not read; the scans that read no hub, each compared
byte for byte with the same scan without the Bazel files; a root
above another project; `get` by advisory.
- tests/e2e_bazel_build/maven.rs: the two flows of the commit before
this one, turned over. After the ruleset's pin, `scan --mode hosted
--yes` reports the two jars of the lock and appends the block that
`get` writes; the build prints the patched marker, a second scan
writes nothing and `rollback` restores every file. After a version
bump, with the stale entry in the file, the scan takes it out, pins
the patch of the new release, and the build takes that jar.
Measured on macOS arm64: the five Maven flows pass on Bazel 9.3.2,
8.8.2, 7.7.2 and 7.4.0 at this tree, a flow in 17 to 56 seconds, and
none of the 498 requests behind a logging proxy went to a public Maven
repository. With the four source files as they were, both flows fail on
9.3.2 at the count of scanned packages (0, not 2), and eight of the nine
tests without a Bazel fail; the ninth holds what does not change. With
the stale entries kept from the sweep alone, the version bump flow fails
on 9.3.2: the scan reports `applied` and no rolledBack. Twenty single
changes to the new code each fail a test; a twenty-first, the root of a
bare scan taken as a discovered directory, changes nothing that a run
can show, because a directory that the built-in ignores would drop has
no project below it that they keep. In sixteen checkouts without a hub
of the ruleset (ten without a Bazel workspace file, six in a workspace
of other rulesets), fourteen scan and get commands print the same bytes,
exit with the same code and write the same files as a binary of the
commit before (224 runs). Over the root MODULE.bazel of 22 open-source
Bazel workspaces, the 18 without a hub of the ruleset give no jar and no
warning.
The line budget of the Bazel group follows the code, 2486 to 2568.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…d to the scan Six fixes from a review of the hub scan, each with a test that fails without it. - The socket.yml policy now holds a stale entry of the block as it holds a package of the run. A scan of a root that `patches.ignorePaths` filters took the entry out of that root's MODULE.bazel, and so did a scan whose policy ignores the package: the sweep got the stale pins whatever the policy said. They are filtered first (`ScanPolicy::admits_restore`). - A scan with no package at all ran the sweep and never looked at a restore that left a file half written; it went on to "No packages found" and exit 0, where the same failure on the path with packages stops the run with `lockfile_write_failed` and exit 1. Both paths now share that stop (`stop_half_restored`). - A pin that is stale and whose patch is withdrawn too was handed to the restore twice and reported as two `rolledBack` events. It is taken once, as the withdrawn pin, with every file that holds it. - The bound of 64 MiB of hub locks for one workspace was added for the scan, which now reads the hubs on every hosted run at a Bazel root, but it sat in `Hubs::read` and so also changed `get`, `list`, `vex`, `rollback` and the notices: in a workspace with more locks than that, a `get <purl>` that pinned before declined with `maven_bazel_unreadable`. The bound is the scan's alone now; every other reader takes each lock up to 16 MiB as before. - Past the bound each further lock was still read from disk and then dropped. Now the first lock that does not fit is the last one read. - `get <advisory> --mode hosted` named a Maven hub that was not read for an advisory with no Maven artifact at all. The warning is given only when the advisory has a patch of a Maven artifact. Also held by a test and said in the docs, with no change in behaviour: when a stale entry goes beside a Gradle build, the hosted Gradle pin that the first scan wrote for the same patch stays, as a pin in a pom.xml does. `list` shows it, `rollback <purl>` takes it out, and while it stays the Gradle lane declines a patch of another release of the artifact. The line budget of the Bazel group follows the code, 2568 to 2583. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
main gained the per-run report of which patches a run acted on, the telemetry of a failed scan and the experimental patch mirror. One file conflicted, crates/socket-patch-cli/src/commands/scan/mod.rs, in two places: - main replaced `scanned_ecosystems` by `count_by_ecosystem` where this branch had added `stop_half_restored` beside it: both are kept, the old function goes as on main. - main added the failed scan's telemetry to the stop after a restore that left files half written, which this branch had moved into `stop_half_restored` so that a scan with no package stops there too. The call is now inside that function, for both paths. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
PR SummaryMedium Risk Overview The core adds Docs, CLI_CONTRACT, workflow commentary, loc budget, and broad e2e/unit coverage for scan, stale cleanup, advisory Reviewed by Cursor Bugbot for commit 5052434. Configure here. |
A hosted
socket-patch scanat the root of a Bazel workspace now finds the Maven artifacts that its pinned rules_jvm_external hubs build, patches them in a tree without apom.xml, and takes out an entry of its block that the hub's lock has moved past.Based on main: three commits and a merge of main (at 64657ee), and nothing here waits for another pull request. The third commit holds the fixes of the code review.
Decisions for the maintainer
Three choices in this change are yours to confirm; each is argued in full at the end.
pom.xml, the scan writes whatget <purl> --mode hostedwrites there for a hub's jar: the hosted Gradle pin, or the pom lane's own pin and warnings. A test holds that the two commands write and say the same. No Gradle was run on the result. One consequence, measured: when the hub's lock later moves on, the scan takes out the entry of the block and leaves the Gradle pin of the old release, as it leaves a pin in apom.xml.liststill shows that pin,rollback <purl>takes it out, and until then the Gradle lane declines a patch of the new release (redirect_gradle_version_conflict).rolledBackevent with the reasonmaven_bazel_entry_stale. The alternative is a warning that namesrollback <purl>and leaves the entry. The code review argues for the alternative; the change is as built until you decide.scripts/loc-budget.py); the tests are 1,046 added lines. Taking the stale entry out is most of the implementation that was not in the first form, and the tests grew with the cases beside other builds and the checks that nothing changes without a hub.What the code does today
scantakes its Maven packages from the JVM caches of a project that has apom.xml, a Gradle build or a Scala-tool build. A checkout whose jars only a rules_jvm_external hub fetches has none of those. A hosted scan there knows no package, never asks the API, and prints "No packages found. Run your package manager's install first.", whilesocket-patch get <purl> --mode hostedin the same directory pins that jar with one block inMODULE.bazel. A user has to know and name every purl.When the hub's lock moves on after a pin (a version bump, an artifact removed), Bazel stops at the entry of the old release ("does not generate repository"). Only
rollback <purl>orremovetakes it out.A bare scan covers the project directories below
--cwd, and--cwditself when a lockfile or a build file makes it a project. The root of a Bazel workspace needs neither, so from a root above other projects a person's scan covers those and never the root, while the same scan with--jsonscans the root alone.What changes
MODULE.bazelthat names a lock (maven_install.json), the scan takes the jar without a classifier of each artifact the lock has a sha256 for, does not skip, and lists under a repository (or takes from the Maven local repository). The version is the lock's, which is the one Bazel builds. Such a package counts inscannedPackages, and inlockfileOnlyPackageswithnotInstalledunless a crawl found that version installed. A patch found for it is pinned asgetpins it: one block inMODULE.bazel, and the Bazel build takes the patched jar. A jar whose repository has no known name (a+in its version, say) is a package too; a patch found for it gets the existingmaven_bazel_artifact.rollback <purl>does: under the apply lock, previewed by--dry-run, reported as onerolledBackevent with the reasonmaven_bazel_entry_stale, also on a scan that finds no package. With the last entry the block goes and the file is the one of before its first patch. A pin of the same patch in apom.xmlstays. Without this, a scan after a version bump pinned the new release, reportedapplied, and left a build that still failed at the old entry.get <advisory> --mode hostedkeeps the releases that a pinned hub fetches, through the same code.lock_file, a lock or a tag that cannot be read, amaven_installof aWORKSPACEfile, a block that was changed by hand), the run gives one new warning,maven_bazel_hub_not_read, with the file, the line and the reason, and still looks up the jars of the hubs it read. With no package at all a person reads "No packages found (see the warning)." The warning is given only in a workspace that has to do with the ruleset: a call namesrules_jvm_external, theWORKSPACEfile callsmaven_install, or amaven_install.jsonlies at the root.--cwdfor a project, under the heading== . ==, when it is the root of a Bazel workspace whose hubs have jars, something of them was not read, or the block has a stale entry.get <purl> --mode hostedwrites in that checkout: beside a Gradle build the hosted Gradle pin too (settings.gradleand three files under.socket/gradle/), beside apom.xmlwhat the pom lane pins or says of that pom.get,list,vexandrollbackread each lock up to 16 MiB as on main. A bare scan from a Bazel root above other projects reads the hubs up to three times (to decide whether the root is a project, to collect what the run asks, and to scan the root).socket.ymlpolicy filters, or one whose policy filters the package, leaves the entry.get <advisory> --mode hostednames a hub that was not read only when the advisory has a patch of a Maven artifact.No other run takes a package from a hub: a scan in vendored or agent mode, one whose
--ecosystemsleaves Maven out, a--globalone, and one of a project below the Bazel root find the packages they found before. A run of a project in no Bazel workspace, or in a workspace of other rulesets, is byte for byte the run it was.Contents
Three commits and a merge of main, 12 files, +1621 -146 against main.
tests/e2e_bazel_build/maven.rsrecord what the code does today: the scan finds no package, asks nothing and changes nothing, also with a stale entry in the file, although the API has a patch.socket-patch-core/src/bazel/maven.rs(hub_jars, the lock bound),socket-patch-cli/src/commands/scan/discovery.rs,scan/mod.rs,scan/withdrawn.rs(the sweep takes stale entries)scripts/loc-budget.py(82, 25, 17 and 14)bazel/maven.rstests/e2e_bazel_maven_beside.rs, nine teststests/e2e_bazel_build/maven.rsdocs/ecosystems.md,CLI_CONTRACT.md.github/workflows/bazel-maven.yml;scripts/loc-budget.jsonThe line budget of the Bazel group follows the code, 2486 to 2568. No fixture changes. One existing test line changed: a closure's type annotation in the unit tests of
bazel/maven.rs. No existing expectation changed.What it proves, and where
Everything below ran on one macOS arm64 machine, on this tree unless a line says otherwise.
maven::ine2e_bazel_build, run as the workflow's step runs it, on 9.3.2, 8.8.2, 7.7.2 and 7.4.0: all five flows pass on each, a flow in 17 to 56 seconds. Ina_hosted_scan_of_a_tree_only_bazel_builds,scan --mode hosted --yesreports the two jars of the lock, writes exactly the block thatgetwrites and fetches nothing; the build prints the patched marker; a second scan writes nothing;rollback --yesrestores every file. Ina_version_bump_of_the_patched_artifact, the build stops at the stale entry androllback <purl>takes it out, as before; then, with the stale entry back in the file, the scan takes it out, pins the patch of the new release, and the build takes that jar. None of the 498 requests behind a logging proxy went to a public Maven repository.appliedand norolledBack(run before main's last three commits were merged in).tests/e2e_bazel_maven_beside.rs: the scan as a program and as a person, as a dry run and a second time; a root with a lockfile of its own, alone and above another project; a stale entry with and without a patch of the new release, with no package at all, while a hub is in doubt, and beside a pom that keeps its pin; a Gradle build and a pom beside the hub, each compared with whatget <purl>writes and says there; a pom that selects another release; each way a hub is not read; the scans that read no hub, each compared byte for byte with the same scan without the Bazel files; a root above another project;getby advisory.bazel::maven: one table of nineteen workspaces for what counts as a hub's jar, what is said of a hub that was not read and where nothing is said, and which entry is stale; one test that a lock is read once and within the bound.MODULE.bazelof 22 open-source Bazel workspaces, the 18 without a hub of the ruleset give no jar and no warning. Bazel's own gives 146 jars and the warning (amaven.artifactinside a comprehension), rules_jvm_external's 928 jars and the same warning, rules_kotlin's 56 jars, rules_scala's 13 jars and the warning for amaven_installof itsWORKSPACE.-D warningson the workspace and on the real-Bazel test target;cargo checkon all targets; the wholesocket-patch-coreunit suite (8341) and the CLI unit suites (1028); 26 CLI test targets around scan, get, Maven and the hosted engine (1060 passed);e2e_bazel_buildwithout Bazel;cli,spawn_env_hygieneand five ratchets; the script tests (422);actionlintand the tool-pin check on the workflow; formatting of the six changed Rust files.Code review
/code-review highgave ten findings. Each was checked by reading the code, and where it could be run, by a run. Six are fixed in the third commit, each with a test that fails without the fix (six single-line changes were run; each fails its test).Fixed:
socket.ymlsaid, so a scan of a root underpatches.ignorePaths, or one whose policy ignores the package, rewroteMODULE.bazel. The policy now holds them as it holds the run's packages.lockfile_write_failedand exit 1. No test forces such a failure.get,list,vexandrollback, where a workspace with more locks than that would newly decline withmaven_bazel_unreadable. It is the scan's alone now.get <advisory>for a patch of another ecosystem named a Maven hub that was not read. It no longer does.Not fixed, with the reason:
Checked on the merged head, on macOS:
-D warningson the workspace and on the real-Bazel test target,cargo checkon all targets, formatting of the changed Rust files;socket-patch-coreunit suite ran on the review commit before the merge (8,341);e2e_bazel_maven_beside(24 tests, 10 of them this pull request's),e2e_bazel_buildwithout Bazel,cli,spawn_env_hygiene,get,scanande2e_socket_yml_policy: 678 passed, none failed;maven::on real Bazel 9.3.2 behind a logging proxy: the five flows pass, and none of the 165 requests went to a public Maven repository.Not covered
bazel-maven.yml.get <purl> --mode hostedpins the purl it is given and leaves a stale entry of another release where it is, so the build still stops there until a hosted scan,rollbackorremoveran. That is whatgetdoes on main.maven.overridethat another Bazel module declares, a hub that the root only takes withuse_repoand other modules fill (a scan there finds no package and says nothing), amaven_installof aWORKSPACEfile (named, not read), a jar with a classifier or of another packaging, and a jar that reaches Bazel without a hub.get <purl>writes there.MODULE.bazelthat cannot be written) is not tested; it gives the sweep's existing warning. Neither is one that leaves a file half written, which stops the run.maven, a lock with an artifact of another Bazel module, and every stale-entry case but the version bump are held by the tests without Bazel only.mvn,gradleorsbtwere not run: no tool may ask a public Maven repository on the machine used.Decisions for the maintainer, in full
== . ==section for the root. That section is a full scan of--cwd, so it also counts the packages a crawl from the root finds installed in the projects below. The alternative is to leave the root out and say in the docs thatsocket-patch scan .reaches it. I recommend keeping it: without it a person's scan misses the hub silently, and the same scan with--jsondoes not.rollback <purl>and leaves the build stopped. I recommend the removal: a scan that exits 0 should not leave a build that fails.pom.xml. The alternative keeps a jar that only the hub supplied out of those lanes; it needs a marker through the hosted engine, and a fresh clone and a machine with warm caches would then write different files. I kept whatgetdoes.maven_bazel_hub_not_readcomes on every hosted scan at a Bazel root of a workspace that has to do with the ruleset while anything of the hubs is in doubt, whether or not a package has a patch. Bazel's own repository and rules_jvm_external's get it for one tag inside a comprehension. It could be held back until a patch is found. I recommend it as built: the scan says what it did not look at.get. A scan makes it visible in a large workspace: every such patch isskippedwith a warning. Pinning per hub that was read is a separate change.🤖 Generated with Claude Code