MARK

We are finally getting around to moving some of our older Objective-C tests from x86_64 to ARM64 and ran into a crasher that only seems to show up on Apple Silicon and only happens about 1-2% of runs on our CI machines. We haven’t been able to repro it locally at all. These tests are pretty old and have some serious https://ocmock.org/ dependencies that we haven’t cleaned up. It seems to be happening when we are deallocating a mock and the crash log records it as a EXC_BAD_ACCESS (SIGSEGV) - KERN_INVALID_ADDRESS (possible pointer authentication failure).

DAVE

“Possible pointer authentication failure?” Are you doing something crazy with ARM64e?

MARK

No ARM64e at play.

DAVE

Phew. Sounds like a threading problem. Have you tried running it against tsan?

MARK

We’ve tried all the sans: tsan, asan, msan, ubsan, and even comic sans with no luck. Honestly this test has pretty much only one thread running.

DAVE

OCMock is compiled with Automatic Reference Counting (ARC) disabled. Have you tried some of the “classic” tools like libgmalloc?

100

MARK

At this point we are pretty convinced that it is not a memory smash. The stack and heap seem fine.

DAVE

OK, so we’ve got an unpredictable bug that is not a threading bug. Any signals at play?

MARK

No sign of any signal handlers.

DAVE

OCMock does some crazy magic under the hood. Can you remind me of some of the voodoo it do so well?

MARK

What evils doesn’t OCMock do? Did I mention we have our own special version of OCMock that has deviated from the opensource? Disclosure: all things discussed about OCMock below are our own fault.

DAVE

No need to remind me, as most of that was my doing. However it has been a while since I dug through the code, so can you jog my memory regarding some of the fun things it does?

MARK

Let’s see… Off the top of my head mock objects: inherit from NSProxy, make heavy use of associated objects, manipulate the class hierarchy by creating and destroying classes on the fly, dynamically change the size of instances using realloc in the -init method, use message forwarding all over the place and have not one but two APIs for doing all of this. I think it goes without saying that it doesn’t pass App Review.

95

DAVE

Righto. Seems like there may be some fertile ground for bugs in there. Getting back to that stack trace, what is it doing when things go boom?

MARK

The mock object is attempting to release one of its member variables (remember non ARC here!).

DAVE

So there’s this fun little toy called NSZombieEnabled that will tell you if OCMock is overreleasing things.

MARK

It is a zombie, but we went through the code and all of the retains and releases seem to be right. The member variable is basically only initialized in -init and released in -dealloc.

90

DAVE

You can turn on MallocStackLogging and use malloc_history to find out where the object was created and destroyed.

MARK

The stack traces showed us that it was created once in -init, but was destroyed in two separate calls to -dealloc.

85

DAVE

Woah.. does OCMock do something funky with -init?

MARK

Yeah, there are cases where mock objects can get -init called on them twice if folks are stubbing -init and/or +alloc.

DAVE

How do we know if we are being initialized a second time?

MARK

Our mock objects do this fun thing where they store their instance variables in an associated object using objc_setAssociatedObject and a unique key. In -init we check for the associated object, and if we have one, we know that we’ve already been initialized so we can short circuit and just return self.

80

DAVE

Are we stubbing -init or +alloc anywhere?

MARK

No, but to save a question or two, -init is short circuiting in the cases that crash.

75

DAVE

So that explains our crash and the over-release, but now we have a new problem. So we are creating a brand new object using standard alloc/init, immediately checking to see if it has an associated object with a unique key, and occasionally it returns something? That’s beyond odd. How do associated objects work?

MARK

We cracked open the libobjc sources and found that each process has a singleton map of “owning” objects to a map of keys to the “associated” objects. These maps are implemented using a DenseMap “borrowed” from LLVM. Just to make things really fun they use a “disguised pointer” to put things in the map.

70

DAVE

A singleton map? I assume it has a lock around it to protect it from the perils of multithreading?

MARK

Yes sir… locked up tighter than Fort Knox.

DAVE

How old is that LLVM code?

MARK

Pretty old. From poking around in Github it looks like 2019.

65

DAVE

Any chance there have been any fixes since then with something like “fixed major hashing bug in Dense Map” in the title?

MARK

No, no major changes to DenseMap that we can see, and we assume LLVM and clang would have sussed out any problems in DenseMap long ago.

60

DAVE

OK, so back to associated objects. What happens if we call objc_removeAssociatedObjects in -init just before we call objc_getAssociatedObject? That is supposed to restore us to a “pristine state” according to the docs.

MARK

objc_getAssociatedObject still returns an object.

55

DAVE

Huh? Obviously Apple and I have different definitions for the word “pristine”. So we clear out all the associated objects and we still get an associated object? What horror is this? What do the sources for objc_removeAssociatedObjects look like?

MARK

Well objc_removeAssociatedObjects checks if the object has associated objects, and if it does it removes them.

50

DAVE

So the name fits. How does it know if an object has associated objects?

MARK

It appears that the check for associated objects is an optimization that only seems to be at play when using non pointer ISAs (pronounced “is a” as in this object “is a” instance of this class).

45

DAVE

Non pointer ISAs? It’s been a looooong time. Can you give me the TL;DR on ISAs again?

MARK

At the dawn of (32 bit Objective C) history, an ISA pointer pointed to the class object for an instance. When we went 64 bit, Apple runtime engineers got excited with all the room they had and started shoving all sorts of things into the ISA including some bits to track weak references, associated objects and reference counts. You will be shocked to find out that these fancy ISAs are called non-pointer ISAs because they aren’t really pointers. You can actually disable them at runtime by setting the environment variable OBJC_DISABLE_NONPOINTER_ISA to 1.

40

DAVE

Cute. So if we do the fancy bit twiddling to look at the “associated objects bit” for our object what do we see?

MARK

It’s 0 before the call to objc_removeAssociatedObjects, 0 after the call to objc_removeAssociatedObjects, and 0 after the call to objc_setAssociatedObject.

35

DAVE

The first two I expected, but the third one? When does this silly bit get toggled?

MARK

It gets toggled in objc_setAssociatedObject if and only if this is the first time something is being associated with the object.

30

DAVE

Which I assume it figures out by querying the singleton DenseMap that we already established is locked down tight. What happens if we manually set the associated objects bit before we call objc_removeAssociatedObjects?

MARK

Then it becomes 0 after the call to objc_removeAssocatedObjects and switches to a 1 after the call to objc_setAssociatedObject.

25

DAVE

Finally we are almost back to some sanity. So something is not cleaning itself up from the global associated map correctly. Who is responsible for tidying it up?

MARK

Well, objc_removeAssociatedObjects will do it (assuming the bit is set correctly). The other obvious place is any object you deallocate removes itself from the map.

20

DAVE

Is this some weird side effect of forgetting to call [super dealloc] somewhere?

MARK

Good thought, but no, objc_destructInstance is called after -dealloc has been called on the object, and the docs clearly state that objc_destructInstance “removes any associated references this instance might have had.”

DAVE

I assume objc_destructInstance checks the magic bit before attempting to remove it from the map?

MARK

Yes.

15

DAVE

So we are either looking for some random neutrino that just happen to toggle this one specific bit in a complex structure in multiple different objects *or* we have a type of object that doesn’t deallocate properly.

MARK

Seems that way.

10

DAVE

Just to check, does OCMock ever do anything fancy with the ISA pointer?

MARK

This is the one place that even OCMock is afraid to touch.

DAVE

OK, so I’m going to take “types of objects that don’t deallocate properly” for $200 Mark. What types of things are we assigning associated objects to in our OCMock codebase?

5

MARK

Well, we mentioned mock objects already. We also associate an object with the classes that we create to mark them as “classes created by OCMock”.

DAVE

Hold on.. How are those classes created and destroyed?

MARK

Using objc_allocateClassPair and objc_disposeClassPair of course.

0

DAVE

What happens if we create a class pair, associate an object with the class and then dispose of the class pair? Does the object get destroyed?

MARK

Well heavens to betsy, the object is still alive and kicking.

DAVE

Aha! The jig is up! Classes don’t clean up their associated objects. So the chain of events that has to happen is that a class pair must be created, an object associated with it, the class pair destroyed and then another new object allocated at the same address as the previously existing class pair. In that weird and wonderful case our new object would end up having an associated object, but doesn’t get the associated bit set so you can’t clear it using standard APIs unless you know the exact key. It’s technically not an OCMock bug, but I can’t imagine a lot of other folks are doing anything like this on a regular basis.

MARK

It turns out that it’s a rare enough case that we never saw it pop up on x86_64 and apparently it just happens to be more likely on ARM64. Once you know what to do, it’s an easy repro case. We filled FB17596679 with Apple. It also turns out that it’s “Google OCMock” specific as the OCMock maintainer took the concept of our patch to mark classes as “created by OCMock”, but they dodged a bullet that they never saw coming as they implemented it in a way that didn’t use objc_setAssociatedObject on a class.

DAVE

Nasty.

MARK

Yeah.

80–100

Welcome honored objc runtime Apple engineer to our modest web page.

50–70

You’ve associated with some weird objects in your time.

25–45

Too much Swift coding is making you soft.

5–20

Admit it: your first thought on hearing “isa” was Jar Jar Binks.