no quiero,

The question is complete and you have answered the question correctly. There is only one solution, the one which is given by you. Your analysis is completely right but not complete.

using your notations and equations
5 X + Y + 0.25 Z  =  20
X+ Y+Z = 20

and there is another contrain that X,Y,Z should be positive, non zero integers (1,2,3 etc, not 5.3 and 10.6)

1. X is number of books each Rs 5.  However, it is impossible to buy 4 books since total number of books, note books and pencil should be 20 and he has only Rs 20.

Clever, so valid X could be, X = 1 or 2 or 3

For, X=3 we get Y +0.25 Z =5 and Y+Z=17
The solution X=3, Y=1 and Z=16 (your solution) is a valid solution because all are integers

For, X=2 we get Y +0.25 Z =10 and Y+Z=18
The solution X=2, Y=7.33 and Z=10.66 is NOT a valid solution because Y and Z are not integers, you can not buy 7.33 notebooks

For, X=1 we get Y +0.25 Z =15 and Y+Z=19
The solution X=1, Y=13.66 and Z=5.33 is NOT a valid solution because Y and Z are not integers