To distract someone is to drag away their focus: to pull away their attention. When you're distracted, something has taken your focus and your attention away. You're trying to do one thing, but you get distracted by a different thing.
If something distracts you while you're working, you stop being focused on your work: you stop paying attention to your work. Distractions can make you waste time and make mistakes.
You might be talking with someone, and then get distracted by a noise, a movement, or even a sudden thought. Something has taken your focus away from your conversation.