How to remove duplicate lines in a large text file?

How would you remove duplicate lines from a file that is  much too large to fit in memory? The duplicate lines are not necessarily adjacent, and say the file is 10 times bigger than RAM.

A better solution is to use HashSet to store each line of input.txt. As set ignores duplicate values, so while storing a line, check if it already present in hashset. Write it to output.txt only if not present in hashset.

Java:

// Efficient Java program to remove
// duplicates from input.txt and
// save output to output.txt 

import java.io.*;
import java.util.HashSet; 

public class FileOperation
{
    public static void main(String[] args) throws IOException
    {
        // PrintWriter object for output.txt
        PrintWriter pw = new PrintWriter("output.txt"); 

        // BufferedReader object for input.txt
        BufferedReader br = new BufferedReader(new FileReader("input.txt")); 

        String line = br.readLine(); 

        // set store unique values
        HashSet<String> hs = new HashSet<String>(); 

        // loop for each line of input.txt
        while(line != null)
        {
            // write only if not
            // present in hashset
            if(hs.add(line))
                pw.println(line); 

            line = br.readLine(); 

        } 

        pw.flush(); 

        // closing resources
        br.close();
        pw.close(); 

        System.out.println("File operation performed successfully");
    }
}

原文地址：https://www.cnblogs.com/lightwindy/p/9650718.html