Hi Greg,
The problem you're seeing here is that "remember=5" is being set in the wrong place. The ESXi 5 STIG is pretty bad on this check and doesn't clearly state where this should be added. Remove it from the requisite line and add it to the sufficient line. Remember is a pam_unix.so setting, not pam_passwdqc.so. That's what causes PAM to behave poorly after you add that setting on the wrong line.
I wrote about it here as well. http://www.vmfieldtips.com/2013/09/03/esxi-5-stig-password-complexity/ Sorry you went so long without an answer.