Upstream repositories

An upstream is another repository that your repository asks when it doesn't hold a file itself. With upstreams, one Artifex repository can serve your own artifacts and everything your builds depend on, so a build needs one repository URL instead of a list.

Only an administrator can manage upstreams.

Add an upstream

An upstream is either a remote Maven repository, named by its URL, or another repository in Artifex, named by its owner and URL name.

To add one, follow these steps:

1. Open the repository and go to Upstreams.
2. Click Add upstream.
3. Under Kind, select Remote repository or Artifex repository.
4. In the Name field, enter a name for people to read, such as Maven Central.
5. For a remote repository, in the URL field, enter the repository's base URL, starting with https://. For an Artifex repository, in the Repository field, enter OWNER/SLUG.
6. Optional: set the credentials and the options described in this document.
7. Click Add upstream.

For an Artifex upstream, you must be able to read the repository that you name, and it can't be the repository that you're configuring.

Order

Upstreams are asked in the order they're listed, and the first one that answers wins. Put the repository that holds most of what you need first, and the slower or narrower ones after it.

To change the order, follow these steps:

1. Open the repository and go to Upstreams.
2. Drag an upstream to its new position.
3. Click Save order.

An upstream that's turned off is skipped. To turn one off without losing its settings, clear Enabled in its settings.

A better way to keep an upstream out of requests it can't answer is a content rule, which skips it before a request is made at all. For more information, see Content rules.

Credentials

For a remote repository that needs authentication, fill in the following:

• User name: the user name to send.
• Password: the password or token to send.

Artifex sends them as HTTP Basic authentication. The password is never shown again; leaving the field empty keeps the stored one. The activity records that a password was set, never its value.

Options

Option	Default	What it does
Timeout	2000 ms	How long to wait for the upstream to start answering. The body can then take as long as it takes. Between 100 and 60000 milliseconds.
Negative cache	300 seconds	How long a "not found" from this upstream is remembered. Between 0 seconds and 7 days.
Accept HTML responses	Off	When off, a 200 OK whose content type is text/html counts as a miss, because it's almost always a login page or an error page rather than a file. Turn it on only for an upstream that really serves files with that content type.
Fetch variant files early	Off	When a Gradle module file arrives from this upstream, fetch every file its variants list right away. See Gradle Module Metadata.
User agent	CodeSoh Artifex	The User-Agent header to send. Set it when an upstream expects a particular one.
Enabled	On	When off, the upstream is skipped.

What happens on a request

When a build asks for a file that the repository doesn't hold, Artifex does the following:

1. Takes each enabled upstream in order, skipping any whose content rules say it can't hold this coordinate, and any that recently reported this path as missing.
2. For a remote repository, requests the file. For an Artifex repository, asks that repository for its own copy, without following that repository's upstreams.
3. For a remote repository, also requests the checksums that the upstream publishes, and compares them with the bytes that arrived.
4. Stores the file and serves it. The next build gets it without the upstream being asked.

A file that comes from an Artifex upstream is served through, not copied, so one copy answers for both repositories. The repository's file count and storage don't grow.

Access is decided by the repository that was asked, not by the upstream. Anyone who can read a repository can read what its Artifex upstreams supply through it, so don't make a repository public while it has an upstream that holds something private. You have to be able to read a repository to add it as an upstream, but that check happens once, when you add it.

Cached files are marked as coming from an upstream on the file page, together with the URL that they came from and which checksums were verified.

Checksums from an upstream

If the upstream publishes a checksum that doesn't match the bytes it sent, the file isn't kept and the request answers 503 Service Unavailable with a message naming the algorithm. It's reported as unavailable rather than missing, because the file exists and something is wrong with the transfer or the upstream.

An upstream that publishes no checksums is still used. The file page shows which algorithms were verified, which is none in that case.

Missing versus unavailable

What the upstream did	What Artifex does
Answered 404 or 410	Treats the file as missing here, and remembers that for the negative cache duration
Answered 200 with an HTML content type, and Accept HTML responses is off	Treats the file as missing, as above
Answered another error status, timed out, or couldn't be reached	Reports the upstream as unavailable
Sent bytes whose published checksum doesn't match	Reports the upstream as unavailable

If every upstream that could answer reports the file missing, the request answers 404 Not Found, and your build reports the dependency as not found.

If none of them could answer and at least one was unavailable, the request answers 503 Service Unavailable with Retry-After: 30 and a message naming the upstream and what went wrong. A build that retries after the upstream recovers resolves normally. This distinction matters: a temporary outage doesn't get recorded as a missing dependency.

Maven Central

Maven Central (repo.maven.apache.org and repo1.maven.org) answers requests from Artifex with 403 Forbidden, whatever user agent the upstream is given, so an upstream that points at it reports as unavailable. Other public repositories answer normally. To resolve Central artifacts, keep Central in your build's own repository list beside the Artifex repository.

Negative caching

A "not found" is remembered for each upstream and path, for the upstream's negative cache duration. Builds ask for files that don't exist all the time, such as a source jar that was never published, and remembering keeps those requests from reaching the upstream again.

Set the duration to 0 to remember nothing, which is useful while you're setting an upstream up.

Remembered misses are forgotten in the following cases:

• You change the upstream's settings.
• You add or remove one of its content rules.
• You remove the upstream.
• You click Check again on the upstream.

Use Check again when you know that a file has appeared upstream and you don't want to wait for the duration to pass.

Metadata documents from an upstream are refreshed every 10 minutes rather than negatively cached, so a new version upstream appears within that time.

Pinning

The first time a coordinate is answered by an upstream, the version is recorded as coming from that upstream. Every later request for files and version-level metadata of that coordinate goes to that upstream alone.

Pinning means that one version of one artifact comes from one place. A second upstream that happens to publish the same coordinate with different bytes can't supply part of a version that another upstream started.

Two things change a pin:

• Deploying a file into that coordinate takes it over. The version becomes a deployed one and stops going to the upstream.
• Deleting the repository's files of that version, where the settings allow deleting, lets the next request choose an upstream again.

If a pinned upstream can't be reached, the request reports it as unavailable rather than falling through to another upstream.

Metadata from upstreams

A maven-metadata.xml is merged. Artifex generates the document from what it holds, asks every upstream that may answer for the coordinate, and merges what comes back, so a build sees one version list that covers your own versions and the upstreams'.

Version-level metadata of a pinned snapshot comes from the pinned upstream only, so build numbers stay consistent with the files.

Remove an upstream

To remove an upstream, follow these steps:

1. Open the repository and go to Upstreams.
2. In the upstream's row, click Remove.

Its content rules are removed with it, and its remembered misses are forgotten. Files that were already cached from it stay in the repository and are still served.