text-file

name	text-file
description	Guidelines for handling text files
type	text

Text Files Handling Specifications

Goals

• Safely read text files without memory exhaustion.
• Accurately detect encoding to avoid garbled characters.
• Identify underlying patterns (e.g., Log formats, Markdown structure, delimiters).
• Efficiently extract or search for specific information within large volumes of text.

Encoding & Detection

• Encoding Strategy:
  • Default to utf-8.
  • If it fails, try utf-8-sig (for files with BOM), gbk/gb18030 (for Chinese context), or latin-1.
  • Use chardet or similar logic if encoding is unknown and first few bytes look non-standard.
• Line Endings: Be aware of \n (Unix), \r\n (Windows), and \r (Legacy Mac) when counting lines or splitting.

Inspection

• Preview: Read the first 10-20 lines to determine:
  • Content Type: Is it a log, code, prose, or a semi-structured list?
  • Uniformity: Does every line follow the same format?
• Metadata: Check total file size before reading. If >50MB, treat as a "large file" and avoid full loading.

Querying & Reading (Large Files)

• Streaming: For files exceeding memory or >50MB:
  • Use with open(path) as f: for line in f: to process line-by-line.