Debugging Rust Cargo Issues in Gitlab

Rust plays a key role in my company's backend applications. It's widely recognized for its ability to create safe, fast, concurrent, modern, and efficient programs.

We were early adopters of the Rust ecosystem. Since we don't do open-source, we needed a place to store our Rust packages (known as Crates) for reuse and manage dependencies. We couldn't use the public Crates.io registry offered by Rust, and there weren't any other options at the time. So, a coworker who also served as DevOps wrote a custom registry in the Go language. It wasn't a difficult task for him since it was technically straightforward - just an HTTP server with some API calls and an index stored in a GIT repository. He even added MongoDB for persistence, but in my opinion, it's not necessary.

When Jfrog introduced Cargo registry support on Artifactory, we made the switch. It was a challenging and straightforward process at the same time due to two reasons:

• There was no option to migrate at either the application level (Artifactory) or registry level (Cargo).
• I wrote a script to go through all of our Rust repositories, modify the config in the code to point to the new registry, and run a "Cargo push". However, each repository had a different directory structure and configurations, making it a tricky process to parse.

Alright, let's move on from the preludes. The focus of this article isn't migration, so let's get to the main point.

One day, a developer showed me the following error in GitLab CI:

$ cargo build
Updating `my-cargo-registry` index
error: failed to select a version for the requirement `my-lib-core = "^2.2.1"`
candidate versions found which didn't match: 2.2.0, 2.1.2, 2.1.1, ...

At first glance, the error seemed obvious. So, I checked the Artifactory project and saw that the artifact with version 2.2.1 existed. It seemed like the issue was with the index. I located the Cargo index in the same Artifactory path and found that the record with the correct version was in place within the config.json file.

"Let's push it to Artifactory again," I thought. Thank goodness the repo was still actively maintained. I made the changes at the CI level and verified that the file was updated on the remote. To double-check, I also clicked the "Recalculate Index" button on Artifactory, which has helped me in the past. Additionally, I compared the dates and checksums of this particular file for added assurance.

Despite all these steps, the builds still failed.

Alright, let's try the CI steps on a local machine. This is a standard diagnostic step for any engineer. To get a more precise result, I deleted all Cargo data from ~/.cargo and ran the "cargo clean" command at the project level, and... it worked locally. Great!

I repeated the steps on the CI system, and... still no luck.

Okay, let's delve deeper. The error message was not helpful, so I added "RUST_LOG=debug" in front of the command and changed the command to "cargo update". To my surprise, the log was the same (as mentioned earlier) without any extra information. It seems that the debug option isn't available for the "update" command.

And then I remembered. I had fixed a similar problem with Cargo on this Gitlab CI before. It was related to the Cargo cache (sometimes, Cargo can act up and not see new crates if the cache is corrupted). One of the GitHub issues suggested deleting the ~/.cargo directory (which I did locally). However, there was also a CI cache on the Gitlab side, so I had to clear it and temporarily turn it off.

Gitlab template with cache:

.rust_build_cache:
  cache:
    key: "$CI_PROJECT_NAMESPACE-$CI_PROJECT_NAME"
    paths:
      - $CARGO_HOME/registry

I also clicked the Clear Runners Cache button on the Gitlab CI project page.

I tried running the CI job again and again... still had no success.

I spent about an hour jumping from one browser tab to another, reading issues, experimenting locally, and feeling frustrated until I learned that you could set environment variables to override the registry in config files. These variables are called:

CARGO_REGISTRY_NAME
CARGO_REGISTRIES_MY_CARGO_REGISTRY_INDEX

And they were pointed to our old custom registry.

I finally found the root cause of the problem. The CI was using old registry variables that pointed to our custom registry. I missed them at first because they were not in the gitlab-ci.yaml file, but in the CI/CD settings under the project's variables web page. The developer had set them, so I wasn't aware that they could overwrite my changes. This was both obvious and frustrating to realize at the same time.

Straightforward moral of the story: always first check environment variables in CI settings. They can overwrite your configs and cause unexpected issues.

Lead image source.