Well, it seems to have the opposite effect of what I originally thought. If I'm understanding correctly, it doesn't actually subtract away darkness, but adds the inverse of each pixel, right?
If that's the case, I guess it's back to square one figuring out an efficient way for getting rid of darkness. I know it probably has something to do with masks, but using pictures gets messy whenever light areas overlap. Any advice on how to achieve something like this?
Unfortunately, that would, to the best of my knowledge, require a dynamic alpha mask for most uses. Which rpg maker does not support natively. The only real way to add alpha masking to an RM Game would be to create a .dll file that dynamically modifies the image data in memory. The problem with that is, the original image data is NOT preserved. At all. I've worked with this in the past (I even created code that almost perfectly mimics the default RM* transition system, including the blending option), and if you need the original image back at some point, you have to clone it before you do any work on it. Which would mean you would have to recalculate the entire image every time a light turned off. Which isn't particularly effective, but, seeing as it's in C or C++, at least it's faster than doing it in Ruby.
If you're interested in a basic .dll file that can modify the image stored in RM* memory, I'm pretty sure a properly targeted search through my posts will turn up at least a couple posts of mine with such code, as well as several posts of mine where I link to said posts with code. Though, I never did post the final code for that transition.dll, now that I think about it. I'll have to see if I still have it somewhere.
Edit: I figured I should also clarify exactly what that blend type does. In the case of subtraction, it will take an image and, if the inverse of the image's color would make a pixel darker, it will essentially merge the two. If it would make it lighter, or if it's transparent, or if the overall value (essentially brightness) remains the same, then it makes no difference and doesn't change it. This is in opposition to the addition blend mode, where if the pixel is brighter, then it will change it, but if it's the same or darker, it won't. Colors do play a role in both of these, as you would see if you did something like setting a sprite to subtract mode. It actually looks kinda neat, and can be used to make a nice kind of evil ghost effect. Addition can make a nice good ghost effect, too.