Stick with the AP class. You don't have a flawless 4.00, but don't throw away everything you built this into. If you don't soldier on and complete the AP, this course was for nothing.
Are you sure you can't take the AP Physics exam next semester? If you score high enough on that, it should reflect that you fought to learn the material despite a rough first semester.