Regarding the term "Panchamas", I quote some of the sutras from Manava Dharma Sastra (aka Manu Neethi).
Talking about the classification of offsprings born of mixed caste marriages, Manu says:-
निषादस्त्री तु चण्डालात् पुत्रम् अन्त्यावसायिनम् । श्मशान गोचरम् सूते बाह्यानाम् अपि गर्हितम्॥ १०-३९
10.39. A Nishada woman bears to a Chandaala, a son (called) Antyaavasaayin, employed in burial-grounds and living within its bounds, and despised even by those excluded (from the Aryan community, i.e., the four castes brahmana, kshatriya, vaisya and sudra).
Describing the vaisvadevam ritual, Manu states, among various other details, as under:-
शूनाम् च पतितानाम् च श्वपचाम् पापरोगिणाम् । वयसानाम् कृमीणाम् च शनकैर्निर्वपेद्भुवि ॥ ३-९२
3.92. Let him gently place on the ground (some food) for dogs, outcasts, chandaalas (svapaakas), those afflicted with diseases that are punishments of former sins, crows, and insects.
Note: Thus chandaalas and svapaakas (another outcaste group beyond the pale of chaaturvarnya); since the place of the offering was just outside the Brahmana's house, and since the chandaalas, svapaakas, etc., could never aspire to come even within the Aryan (four caste) village, this sloka just tells the reader that the status of the chandaala/svapaaka/outcast was nothing more than that of the dogs.
Among the rules of conduct for a good brahmana, Manu lays down as under:-
न सम्वसेत् च पतितैर्न चाण्डालैर्नपुल्कसैः । न मूर्खैर्नाविलप्तैश्च नान्त्यैर्नान्त्यावसायिभिः ॥ ४-७९
4.79. Let him not stay together with outcasts, nor with Chandalas, nor with Pukkasas, nor with fools, nor with overbearing men, nor with low-caste men (antyaas), nor with Antyaavasaayins. Here Manu mentions "antyaavasaayins" as different from "antyas" or low-caste men.
In the section dealing with offsprings of the offsprings of inter-caste marriages, Manu says,
यथैव शूद्रो ब्राह्मण्याम् बाह्यम् जन्तुम् प्रसूयते । तथा बाह्यतरम् बाह्यश्चातुर्वर्ण्ये प्रसूयते ॥ १०-३०
10.30. Just as a Sudra begets on a Brahmana female a being which is excluded (from the Aryan community), even so (a person who is himself so) excluded procreates with (females of) the four castes (varnas), (sons) who are more (worthy of being) excluded (than he himself).
This is the origin of a group or caste or whatever one might call them, as the "fifth caste" (baahyaas). They were not allowed to live near any of the four castes and the burning place (smasaanam) was earmarked for them, outside the village, settlement of the Aryan community.
The groups antyaavasaayins and antyas got combined, in course of time, since both were outside the pale of Chaaturvarnya and Aryan settlements, and came to be denoted by the common word "antyaja".
These lowest group of human beings, cast outside the "chaaturvarnya" system of the Aryans, were generally known as "antyaja" as stated above. In Madras Presidency, they were originally grouped as "Paraiyas and kindred classes", which included (according to earlier references) the Pulayan, Chakkilian, and Thotti. This classification was replaced, probably for convenience, by the term "Panchamas" first in the Madras Census Report, 1871. Obviously, it must not have been an invention by a Britisher but of a native who was aware of this term being applied to denote the said classes. Since the Madras Presidency included, besides the present day T.N., other areas also, similarly disadvantaged groups or classes from the other areas also got included in the term "Panchamas". AFIK, this term was not current outside the Madras Presidency. In the north terms like "Dom, Domb" etc., were used, I think.