The fastness of CSV and text file depends on the use of it. Deep down both CSV and text file store data in the same way on memory.
Text file store data with no rules and standard format it directs store string as plain text. And another hand CSV file stores data in standard formate as rows and columns.
the column is generally separated by a comma but it is ok if we use other character value. The full from of CSV is Comma separated value but also mean character separate value.
so at the time of retrieve data from the file, we have a specific address for specific data so it is easy for retrieve data like name, age, address, id or many more we can imagine.
In CSV file we can able to store categorical of data because of it formate which is not possible in the text file so in case of categorical data CSV is faster than text.
And if we don't have categorical data and has single formatted data on the whole text file so it is easy to retrieve so in this case text file is faster than CSV file. but the text must have only one kind of data like a paragraph of the book.