USCIS website says: " If you hold refugee or asylee status and are not a permanent resident, you must have a refugee travel document to return to the United States." I did not find where it says you need travel document if you are LPR.