don't worry if you have not been caught at working.

If you are not caught, they don't know you worked!!!

you shouldn't have problem at port of entry just because you worked in summer job using your SSN unless... you use your old paystubs to show your source of income to support your study. I don't think you are that dumb as you're already enrolled in a reputed US college.