Codex Cloud has severe issues with hex.pm (Elixir package manager)

Coding with ChatGPT
1

Codex can’t properly fetch dependencies for Elixir mix projects (like Phoenix). Doing a simple mix deps.get is causing all kinds of issues with the proxy that the Codex Cloud VM is using.

Elixir is a popular programming language and currently to get it working with Codex, we need to use alternative mirrors, and in my tests, the only one that sometimes works is the one hosted in China which is not idea.

I wasn’t able to find where to report feedback or bugs, so I’m writing it here. I hope the team can look into this and tweak the Codex VM environment a bit to make Codex usable for Elixir development without a huge bootstrap script.

Findings

  • Running mix deps.get fails immediately because Mix cannot download deps from builds.hex.pm, reporting an HTTP 503 from the upstream service through Envoy.

  • Even installing Hex explicitly with mix local.hex --force encounters the same 503 response, so the failure happens before any project dependencies are fetched.

Network observations

  • The container is configured to use a corporate HTTP(S) proxy (Envoy MITM) for all outbound traffic, as shown by the numerous *_proxy environment variables and the injected CA certificate paths.

  • Direct access with curl succeeds (HTTP 200), so the proxy itself can reach builds.hex.pm when acting on behalf of a typical client.

  • When the Erlang/OTP HTTP client (:httpc) tries to reach the same endpoint without proxy settings, the call fails with :enetunreach, showing that direct egress is blocked; the proxy must be used.

  • After pointing :httpc at the proxy, the request reproduces the 503 “upstream connect error … connection termination” returned by Envoy, matching Mix’s behavior. This indicates an incompatibility between the Erlang HTTP stack that Mix uses and the environment’s proxy handling for HTTPS traffic.

Conclusion

Mix relies on Erlang’s built-in :httpc, which either cannot reach hex.pm directly (egress blocked) or, when routed through the mandated Envoy proxy, receives a 503 because Envoy terminates the upstream TLS connection before headers are returned. That proxy-level failure prevents Mix from downloading Hex, explaining why **mix deps.get**cannot proceed without switching to an alternate mirror or otherwise adjusting the proxy configuration.

For completeness sake, here is the script I came up with that kinda works to install elixir dependencies by manually installing rebar, and then using the china hosted UpYun Hex mirror:

#!/usr/bin/env bash
# Installs Erlang/Elixir and project dependencies using a Hex mirror to avoid 503 errors.
set -euo pipefail

# Install required runtimes via mise and activate shims
mise install erlang@27.2 elixir@1.18.4-otp-27
eval "$(mise activate bash)"

# Ensure Hex and Rebar are available
mix archive.install github hexpm/hex branch latest --force
curl -fSL https://github.com/erlang/rebar3/releases/latest/download/rebar3 -o "$HOME/.mix/rebar3"
chmod +x "$HOME/.mix/rebar3"
mix local.rebar rebar3 "$HOME/.mix/rebar3"

# Use UpYun Hex mirror and relaxed HTTPS to bypass 503s
export HEX_MIRROR=https://hexpm.upyun.com
export HEX_UNSAFE_HTTPS=1
export MIX_REBAR3="$HOME/.mix/rebar3"

# Fetch Elixir dependencies
mix deps.get

# Download Tailwind binary using GitHub token to avoid 503s
TAILWIND_VERSION=4.1.7
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m)
case "$ARCH" in
  x86_64|amd64) ARCH=x64 ;;
  aarch64|arm64) ARCH=arm64 ;;
  armv7l) ARCH=armv7 ;;
esac
TARGET="${OS}-${ARCH}"
if [ "$OS" = "linux" ] && command -v ldd >/dev/null && ldd --version 2>&1 | grep -q musl; then
  TARGET="${TARGET}-musl"
fi
TAILWIND_DEST="_build/tailwind-${TARGET}"
TAILWIND_URL="https://github.com/tailwindlabs/tailwindcss/releases/download/v${TAILWIND_VERSION}/tailwindcss-${TARGET}"
mkdir -p "$(dirname "$TAILWIND_DEST")"
if [ -n "${GITHUB_TOKEN:-}" ]; then
  curl -fSL -H "Authorization: Bearer $GITHUB_TOKEN" "$TAILWIND_URL" -o "$TAILWIND_DEST"
else
  curl -fSL "$TAILWIND_URL" -o "$TAILWIND_DEST"
fi
chmod +x "$TAILWIND_DEST"

Funnily enough I told Codex to find a solution and it built 750 LoC python proxy that mints a trusted local CA, terminates TLS, removes invalid hop-by-hop headers, and replays Mix’s HTTPS requests through the existing upstream proxy so Hex endpoints succeed behind Envoy. Kinda works but I prefer to not rely on that :laughing: