16 17: They stack.
Per the SRD, regarding stacking
In most cases, modifiers to a given check or roll stack (combine for a cumulative effect) if they come from different sources and have different types (or no type at all), but do not stack if they have the same type or come from the same source (such as the same spell cast twice in succession).
Not only are vicious and merciful untyped (and additionally come from different sources), they are not modifiers to a roll. They each offer additional dice as bonus damage.
17 18: All damage is non-lethal.
Merciful states that "The weapon deals an extra 1d6 points of damage, and all damage it deals is nonlethal damage.
" This is not overwritten by the vicious special ability, or any other mentioned special ability.
Yes, it retains its type, even though it is non-lethal damage, as the merciful special ability does nothing to change type of damage (note that it does not affect bludgeoning/piercing/slashing either for DR purposes).
Yes, Vicious damage to the wielder is non-lethal as well, as the vicious special ability does not overwrite merciful's ability that all damage is nonlethal.