In their website, they said you "must" have RTD if you are in refugee or asylee status and you "may" have RTD if you are a permanent resident. RTD works like your identification documents. So when I asked USCIS, they said you don't fall under asylee/refugee status once you are a permanent resident. Like @fdpower mentioned, if you don't have pp from your country eventhough you are a permanent resident, then you require a RTD. As per the phone conversation with USCIS, that person told me to put down name, his/her agent id, date and time of phone conversation if asked by officers at POE.