The only experience I have with a friction modifier is in my mustang. The rear diff requires it or the tru-trac can pop while turning at low speed. Before you can understand how friction modifiers function, it is important to understand how "wet" clutches perform.
In a wet clutch arrangement, there are three stages of engagement. During the first stage, the clutch is not in contact with the pressure plate or other metal plate. We will use an automatic transmission torque converter as an example. Anyone who has driven auto trans cars with lockup torque converters for a while has probably experienced a phenomenon known as "lockup shudder" or "torque converter shudder". Shudder is caused during torque converter clutch lockup by burnt fluid or fluid wich has exhausted all of it's friction modifers. The result is a chattering feeling when the torque converter goes into lockup mode. I will now attempt to explain the physics of wet clutch engagement.
As I have already mentioned, during the first stage the clutch is not in contact with it's mating surface. The fluid itself, however, is acting as a viscous coupling, causing a partial engagement. A side effect of this is heat, and I believe you all know that heat is the killer of automotive oils.
The second stage is very simlar to the first. At this point the clutch is very close, possible within thousandths of an inch, from it's mating surface. The viscous coupling is now more effective, but the pressure and shear load on the fluid are also higher, and the result is increased heat.
During the third stage, the clutch actually contacts it's mating surface and positive engagement is reached. The shear load of the fluid has been overcome and has either extruded itself outside of the clutch material or, depending on the application, has partly or entirely extruded itself through a porous friction material, thus exiting the engagement area of the clutch.
Now that we have an understanging of wet clutch engagement, lets see how that plays out in the real world. If a fluid has lost a substantial amount of the friction modifier, the shear of the fluid will be inconsistant accross the engagement surface and the clutch will briefly alternate between full engagement (stage 3) and viscous engagement (stage 2). The as power through the assembly varies, wich is connected in our case to a vehicle that we are inside of, a bucking of sorts is perceptable to it's occupants as power transmitted to the wheels is momentarely interrupted and regained.
In an application like a limited slip rear end, similar phenomena occur but in a slightly different manner. Because the clutch plates are constantly loaded with heavy springs, in theory they should always remain in stage 3 of lockup. If that were the case however, they would never slip. So the purpose of a friction modifier in a rear end is to ensure that the transition from S3 to S2 and back again during cornering etc. is smoothe. Therefor chatter occurs in much the same way it does in our torque converter clutch scenerio.
So the answer to your question if friction modifiers enhance friction or reduce it? The answer is there is no answer. Depending on what the application calls for and how it is engineered, they can do either. So as Alex said, it does just that; modifies friction.