Catalina is checking notarization of unsigned executables
May 22 2020 by Jeff Johnson
Support this blog: Link Unshortener, StopTheMadness, Underpass, PayPal.Me
This is a follow-up to Allan Odgaard’s excellent article macOS 10.15: Slow by Design. I want to talk specifically about the first section “Spawning a new Process”, because there has been widespread misunderstanding of this. Odgaard provides a simple test to show that the first run of an executable is delayed while Catalina checks the executable’s notarization status online. This occurs even for shell scripts, which cannot be code signed!
echo $'#!/bin/sh\necho Hello' > /tmp/test.sh && chmod a+x /tmp/test.sh
time /tmp/test.sh && time /tmp/test.shA number of people, including myself, have reproduced this issue. However, a number of other people claim that they can’t reproduce it. I’m rather skeptical of these claims, for I’ve found more than a few cases of people misinterpreting their own tests. You can’t expect to just experience an obviously long delay, because the length of the delay depends crucially on the condition and speed of your network connection, as well as your vicinity to Apple’s servers. (A Chinese commenter on Hacker News posted results of 5.746 seconds vs 6 milliseconds on VPN and 1.936 seconds vs 5 milliseconds non-VPN.) Without an online check, the script ought to finish executing within a few milliseconds. So even if your first test result is only around 100 milliseconds, that’s still many times longer than expected. Some people try to explain away the delay, e.g., “I would put the 300 vs 5 ms down to filesystem caching”, but such hand waving doesn’t stand up to further scrutiny.
It’s also important to remember that Catalina caches the result of the online check. If you test multiple times, this could give misleading results. In my testing, the easiest way to trigger a new online check is to modify the contents of the script. I simply created a script in TextEdit, saved it, and then kept TextEdit open for easy editing and saving.
You can verify that there’s an online check by taking packet traces. On the first run, there’s clearly an online check. On subsequent runs of the unmodified script, there’s no online check, and the execution is much faster. If you turn off your Mac’s internet connection, you can see that the first run of a script is now much faster than before; yet the first run without internet is still slower than subsequent runs. So it’s still trying to make an online check, but failing, and then caching that result.
Is Catalina trying to check the notarization of the executable? The evidence strongly indicates yes. The packet traces for this look exactly like the packet traces for a normal app notarization check. They’re always to the same domain: api.apple-cloudkit.com. If you search the Catalina file system, there are only a few results for this domain. I found /System/Library/PrivateFrameworks/WorkflowKit.framework and /System/iOSSupport/System/Library/PrivateFrameworks/WorkflowKit.framework, which are related to Shortcuts, /usr/libexec/remindd, which is related to Reminders, and /usr/libexec/syspolicyd, which is what we’re looking for, the process responsible for checking notarization. In fact, there’s only one place — https://api.apple-cloudkit.com/database/1/%@/%@/public/records/lookup — where syspolicyd contacts that domain. (The %@ in the URL are Objective-C format strings. Yes, syspolicyd is still written in Objective-C, as is the majority of macOS.) You can also find the log message in syspolicyd “Performing legacy notarization check for unsigned code”. I’m not sure why it says “legacy”; I could speculate, but that would just distract from the main point. In any case, I don’t know how this could plausibly be called anything other than a notarization check, because there’s only a single online API here, which is used for both regular apps and non-app executables.
By the way, you can block macOS notarization checks without turning off your internet connection by installing Little Snitch and setting the rules to deny any outgoing connection from syspolicyd. I highly recommend Little Snitch and consider it to be absolutely essential software for the Mac.
In contrast to Catalina, macOS 10.14 Mojave has no online check whatsoever for shell scripts. This can also be verified by packet trace. Even if I download a file from the web, adding a quarantine extended attribute, there’s still no notarization check. The first run of a shell script on Mojave is as fast as subsequent runs of the script. There’s no online delay introduced by Mojave.
Back to Catalina, I started wondering about other kinds of executable. What about compiled command-line tools that are not scripts but not apps either? I created a simple “Hello World” project in Xcode, and I changed the build settings so that the tool was not code signed at all by Xcode. When I ran the tool for the first time, there was no online notarization check, which was a bit surprising to me. When I looked at the Xcode build transcript, though, I found the explanation. The final phase of the build, after the linking phase, was “Register execution policy exception”. Xcode called builtin-RegisterExecutionPolicyException on my tool. This gave the tool permission to execute on my Mac without getting checked.
Since Xcode was messing with my test, I decided to forgo Xcode, instead directly compiling a command-line executable using clang. I didn’t code sign the compiled executable. When I ran it the first time, there was a notarization check! And of course there was no notarization check on subsequent runs. Thus, Catalina seems to be checking notarization on every unsigned executable, whether it’s a shell script or a compiled Mach-O file.
For my next test, I decided to compile another tool but this time perform “ad hoc” code signing on it (codesign --sign with the identity “-“). An ad hoc signed executable is signed only for your Mac, and doesn’t require a signing certificate. When I ran my ad hoc signed tool the first time, this also triggered a notarization check, like the unsigned case.
The one scenario I didn’t test was running a compiled command-line tool signed with my Apple developer certificate. That’s easy to do with Xcode, not so easy to do without Xcode. It’s not impossible, you just have to figure out the appropriate arguments to the codesign invocation. To be honest, after all of my other testing, I just didn’t feel like doing the additional work. Someone else can try this, please!
Returning to the original article, Allan Odgaard asks, “Are Apple sending the source of all my custom scripts to their server?” The answer is no. There are two independent ways of verifying this. First, you can look at the disassembly (otool -tV /usr/libexec/syspolicyd) to see exactly how it works. Second, you can look at the packet traces. For example, if you create a very large script, you can compare the size of the script with the number of bytes of data sent to Apple. There just aren’t enough bytes to fit a large script; there aren’t many bytes at all in a notarization check, which is the point, because it’s supposed to be a fast as (networkly) possible. Code signing and notarization use a secure hash, not the full file contents.
I hope this discussion has been a useful clarification. Unfortunately, I can’t explain why Catalina is running notarization checks on these executables. That mystery is really up to Apple to explain. You’ve got a lot of explaining to do!
Addendum May 23
Today I came across an intriguing comment by an Apple Xcode engineer. (By the way, this engineer has me blocked on the bird site, for unknown reasons… unknown except for generalized ripping on Xcode, I guess.)
Xcode (the UI) is able to bypass GateKeeper checks for things it builds.
The “Developer Tool” pane in System Prefs, Security, Privacy is the same power. Drag anything into that list you’d like to grant the same privilege (such as xcodebuild). This is inherited by child processes as well.
The point of this is to avoid malware packing bits of Xcode with itself and silently compiling itself on the target machine, thus bypassing system security policy.
So this may indicate Apple’s motivation for adding notarization checks to scripts and other standalone executables. One major problem, though, is that this information is not documented anywhere, to my knowledge. It’s the first I’ve ever heard of it, and I’m a professional Mac developer. As is Allan Odgaard. But we’re all just flailing in the dark, trying to cope with mysterious system behavior.
Hacker News comments are not documentation, Apple!
Let me leave you with another intriguing comment to ponder:
Making this about speed is burying the lede. From a privacy and user-freedom perspective, it’s horrifying.
Don’t think so? Apple now theoretically has a centralized database of every Mac user who’s ever used youtube-dl. Or Tor. Or TrueCrypt.