How to read the last MB of a very large text file
Asked 07 September, 2021
Viewed 2K times
  • 62
Votes

I am trying to find a string near the end of a text file. The problem is that the text file can vary greatly in size. From 3MB to 4GB. But everytime I try to run a script to find this string in a text file that is around 3GB, my computer runs out of memory. SO I was wondering if there was anyway for python to find the size of the file and then read the last megabyte of it.

The code I am currently using is as follows, but like I said earlier, I do not seem to have a big enough memory to read such large files.

find_str = "ERROR"
file = open(file_directory)                           
last_few_lines​ = file.readlines()[-20:]   

error​ = False  

for line in ​last_few_lines​:
    if find_str in line:
    ​    error​ = True

3 Answer