The documents to invite your brother would be the same set you would use to invite your parents. But, assuming ur brother is less than 45, chances of visa is slim. Since it is a marriage ceremony and u want ur brother to be here, i think is ok to take the chance.

Documents needed might be, I-34  affidavit of support, documents to prove ur status, W2 from previous year. letter of employment etc. The burden is on your brother to proof to the visa consular that he will return after the marriage.

Good luck.