Attempt to install incompatible Xcode on Mac build agents

TL;DR

I’ve tried to update our tools to latest ones, despite they weren’t compatible with our Mac machines and a lot of things went bad.

Act 0: Inspiration

Recently there was WWDC 21 where new operating systems were presented. If you know Apple well, they always present new tools and SDKs in June (WWDC) and new iDevices will be presented in September (Keynote). At some point after releasing new iDevices and operating systems, App Store Connect will eventually reject builds with old toolchains. Our CI/CD scripts were using Xcode 12.1 which was released 20 Oct 2020 and was getting older everyday. Especially after release of Xcode 13.0 Beta 1. I was preparing our build agents for updates and installed the latest non-beta release – Xcode 12.5.

Act 1: Horrible realization

Adding new Xcode version was straightforward. Just download xip file from Apple, put it to shared public S3 and give link to provisioning system. Everything went OK until we actually needed to build something. Errors were raising, saying:

Executable requires at least macOS 11.0, but is being run on macOS 10.15.7, and so is exiting.

I forgot, that our build agents were running macOS Catalina, because the server from MacStadium offered us Mac mini (late 2012) which wasn’t supported by macOS Big Sur. And the new Xcode were requesting macOS Big Sur. Every single command invoking Xcode tools were logging this. At first tried to uninstall the New Xcode, but now older tools also were requesting newer macOS.

Act 2: Trying to fix everything

Now I had to SSH every machine. Switched Xcode’s manually by xcode-select -s and it didn’t work. Uninstalled /Applications/Xcode-12.5.app, removed symlinks and reinstalled Xcode 12.1. Now we had different error:

$ xcrun simctl list
dyld: lazy symbol binding failed: can't resolve symbol _rosetta_is_current_process_translated in /Library/Developer/PrivateFrameworks/CoreSimulator.framework/Versions/A/CoreSimulator because dependent dylib #1 could not be loaded
dyld: can't resolve symbol _rosetta_is_current_process_translated in /Library/Developer/PrivateFrameworks/CoreSimulator.framework/Versions/A/CoreSimulator because dependent dylib #1 could not be loaded
zsh: abort      xcrun simctl list

Clearly dyld was trying to use rosetta, which was introduced in Big Sur. Now we have linked to a new system libraries (big F). Now removed every developer files from system:

$ sudo rm -rf /Library/Developer
%

And then I had to install developer command-line tools again:

$ xcode-select --install 
xcode-select: note: install requested for command line developer tools

$ # GUI alerts is presented, but you cannot click on it from Terminal
$ softwareupdate -l
Software Update Tool

Finding available software
Software Update found the following new or updated software:
* Label: Command Line Tools for Xcode-12.4
    Title: Command Line Tools for Xcode, Version: 12.4, Size: 440392K, Recommended: YES,

$ softwareupdate -i -a
Software Update Tool

Finding available software

Downloading Command Line Tools for Xcode
Downloaded Command Line Tools for Xcode
Installing Command Line Tools for Xcode
Done with Command Line Tools for Xcode
Done.

Now we had other errors raising:

$ xcrun simctl list
...
xcodebuild: error: The authorization was denied since no user interaction was possible.
...
/Applications/Xcode-12.1.app/Contents/Developer/usr/bin/simctl: line 15: /Library/Developer/PrivateFrameworks/CoreSimulator.framework/Versions/A/Resources/bin/simctl: No such file or directory
/Applications/Xcode-12.1.app/Contents/Developer/usr/bin/simctl: line 15: exec: /Library/Developer/PrivateFrameworks/CoreSimulator.framework/Versions/A/Resources/bin/simctl: cannot execute: No such file or directory

And now it’s time to reinstall Xcode 12.1. And of course bundle exec doesn’t work. System provided ruby doesn’t work well with bundler, so we have to use Homebrew provided ruby installs. And they don’t install to default search path /usr/lib/bin. Do not forget to set $PATH upon SSHing to machines:

$ export PATH="/usr/local/opt/[email protected]/2.7.3/bin:/usr/local/opt/ruby/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH"
%

Did I mention that every reinstalling Xcode takes about 20-25 minutes? After all this everything finally works.

Summary

I always read release notes for every update. But forgot that our servers (Mac build agents) cannot be upgraded to the new macOS. Automatic checking should be implemented upon installation.

CI/CD wasn’t operational at least for 4 hours (10 Jun, 2021).