put very big content in a string or stringbuilder

I want to fetch a very big html page, however when I tried to use jsoup for parsing the page it reported a lot of erros because the page is too large.

I also saved this page as a text file (resulting in a 225mb file), but the file is so large it exceeds the 2147483647 characters limit of String and StringBuilder.

How can I handle such a large string?

2 answers

  • answered 2018-10-11 19:15 banncee

    Download the file and save it locally. Then use Buffered File Readers to read the file line by line and process it. Reading the entire file into one string seems like a bad idea, given it's size, and you still can't analyze the data efficiently.

  • answered 2018-10-11 19:19 Andreas

    The response is text/plain, not HTML, so don't use jsoup.

    Do a simple HTTP GET, and parse the data as it is being downloaded, one line at a time, in order to minimize memory use. No need to store to disk first.