Meter latency with Scarlett 2i2 interface?

I usually use PCI/FireWire interfaces but I picked up a Scarlett 2i2 for smaller jobs. In Logic I've got samples set to 128, Core Audio enabled and the Scarlett is selected as the input/output device on my Mac (no driver required). For some reason I'm encountering a lot more latency than I expected. Logic says that at 128 samples, it should be about 10ms of latency. No way. I'm getting about 500ms to 1 Second of latency. I have a fast quadcore machine with 16GB of ram so that isn't the problem.



What's going on here?

Meter latency with Scarlett 2i2 interface?