LaVA-Man: Learning Visual Action Representations for Robot Manipulation