Unfortunately, I have the same experience, the silicon buds don't fit well and they hurt the ear. Found that it will become loose when you move about also.
Suggest you use the form buds. That fits perfectly and they are very comfortable. Downside is that it gets dirty.