CLASS: Contrastive Learning via Action Sequence Supervision for Robot Manipulation