Physical Damage
Base Damage
$Base Damage = (ATK_{Attacker} + ATK_{SkillBonus} - DEF_{Target}) Damage Multiplier$
Damage Multipliers and ATK Skill Bonuses depend on what skill you're using (or what skills you have) and what level it is. The actual damage done by an attack is slightly random, but will be around your basic damage.
Attacker's ATK
$ATK_{Attacker} = (POW + ATK_{Weapon}(1 + Mastery Bonuses)) ATK Multiplier$
Mastery skills like Sword Mastery will give a percent bonus to your equipped weapon's ATK, if your equipped weapon is of the right type (Sword Mastery only applies to swords, etc.).
Some skills, like the Rogue's Casual Behavior (行住坐臥) will multiply your entire ATK, weapon and all.
ATK Skill Bonus
Most skills don't give an ATK Bonus, but some will. The Princess' EX skill Nine Tail gives an ATK Bonus of 20.
Exhaust
Minimum and maximum damage will be multiplied by 1.5.
Dagger Fetish, Sword Mastery, and Casual Behavior (行住坐臥)
Dagger Fetish and Sword Mastery are both Mastery skills, and the bonuses from these add up before multiplying them with your weapon's ATK.
Casual Behavior will multiply your entire ATK by its bonus.
For example, with Lv. 10 Sword Mastery (+30%), Lv. 10 Dagger Fetish (+70%), Lv. 10 Casual Behavior (140% ATK) and a dagger equipped, your total ATK will be:
$ATK_{Attacker} = (POW + ATK_{Weapon}(1 + .3 + .7)) * 1.4$
Magical Damage
Base Damage
$BaseDamage = \left({INT_{Caster} \over 2}ATK_{Spell} + {5 \over 2}(1 + MasteryLv.) - {(DEF - INT)_{Target} \over 2}\right)DamageMultiplier$
Spell Attack Power and the Damage Multiplier both depend on the level of the spell you're using. Check here (page in Japanese).
Exhaust
Just like physical damage, minimum and maximum damage is multiplied by 1.5.