Replacing mmap_sem with Finer Grained Locks

Intro
The problem with mmap_sem • The situation described earlier (rue conflict between fault and munmap) is extremely uncommon! . Correct threaded programs normally avoid having their threads race against each other • Multiple threads may run non-overlapping memory operations which are logically independent of each other, but the kernel does not make that
mmap_sem false conflicts . Some threads allocate or free some (large) memory blocks • Some threads access memory they have already allocated . Some threads are getting spawned (thus requiring new user stack alocations) • The sysadmin runs the ps command (which shows the processargs stored in its address space)
Goals for mmap_sem replacement . Contention should ideally only occur between threads manipulating the same memory . May block on locks protecting shared data structures, as long as they are only held for short amounts of time
Supporting progressive replacement . Our mmap sem replacement needs to support both coarse lockers (automatically converted from the current mma sem uses) and fine grained lockers (with an associated address range to avoid faise conflicts) . Making a given locker fine grained does not change its interaction with coarse lockers. The only changing interaction is with other fine grained lockers This greatly facilitate progressive conversion of each locker
Converting one mmap_sem locker . When the locker is converted to fine grained, it needs to protect against shared data structures being concurrently accessed by another fine grained locker • Add new locks as necessary • Ensure they are never held for long
Basic implementation ideas . Use a range locking data structure to represent current and pending range locks • Address ranges may be locked for read or write • Add new lock (mmvma_lock) protecting both the VMArbtree and the range locking structure Ot can be convenient to know the VMA contents before acquiring a range lock...
Page fault path: acquiring range lock • Examine VMA for the faulting address • Determine appropriate locking range for the address and VMA type (i.e., 2MB range around the address in the anon VMA case) • Acquire read lock for the address range • Verify VMA type has not changed Retry with coarse lock if it did
Page fault path: faulting the page • Fault the page as usual, based on VMA attributes obtained at the start of the fault. The attributes won't change as the range is read locked. . Note that VMA attributes have to be looked up at the start of the fault, the VMA can not be referenced later on as a fine grained writer may free it due to VMA merging.
Future plans . Grow the number of places we do fine grained locking • Performance comparison • Expect we may have to add speculative faults to bring the performance up • Possibly replace the centralized range lock with a distributed approach ??? If someone wants to tackle this... (not me)