ITP has a good supply of resistors, capacitors, LED's, and some other basic components and IC's. Once you start researching your own project, you'll need to branch out and start buying boards and components online.
Tinkersphere is just a couple blocks from school, and is stocked perfectly for any p-comp student.
Adafruit creates and documents TONS of hardware projects that we can learn and borrow from. Plus, their in NYC so shipping can be quick.
Sparkfun is similar to Adafruit, except they're in Colorado.
Digikey is where we get the vast majority of our components. Huge catalog that can be confusing at first.
Mouser is another component supplier like Digikey.
Ebay is a great place to find parts and materials you can't find anywhere else. However, depending on where it's coming from, shipping might take a few weeks, so plan accordingly.
Alibaba is like ebay, except it's a lot of stuff from China and it's super cheap! But, once again, things can take a long time to arrive, so plan accordingly.
Other Machine Co. sells useful tools and materials for dealing with PCBs and other things their mill can do.
Metalliferous is in NYC, and has drill bits for making holes, but sadly no endmills to cut sidways. We use #58 for holes, and #70 for vias. I've never tried them, so let me know if you do!
Zephyrtronics has almost everything you'd need for working with SMD parts.