Token reading vs. line reading
Scanner has two reading modes that look similar at the call site but behave very differently. A method that picks the wrong mode reads the wrong characters, leaves the wrong characters behind, and produces output that looks "off by one" with nothing at the call site to show why.
Token mode (next(), nextInt(), nextDouble()): reads a chunk of non-whitespace characters, stops at the next whitespace (space, tab, newline). The whitespace itself stays in the buffer for the next call to consume.
Line mode (nextLine()): reads everything up to and including the next newline, returns the line without the trailing newline. The newline is removed from the buffer.
File contents:
42 17
-3
| Calls in order | Returned values | What is left in the buffer | |---|---|---| | nextInt(), nextInt(), nextInt() | 42, 17, -3 | (empty) | | nextLine(), nextLine() | " 42 17", " -3" | (empty) | | nextInt(), nextLine() | 42, " 17" | newline + " -3\n" |
The third row is the gotcha that produces the "nextLine returns empty string" bug: after nextInt() reads 42, the buffer still holds the space before 17, the 17 itself, the newline, and the second line. nextLine() then reads up to the first newline, which comes after 17. The return is " 17", not "-3". The trap that follows a token-mode read with nextLine carries the full diagnosis.
When to use which
| Use token mode when | Use line mode when | |---|---| | Values are separated by whitespace and you want them one at a time | The unit of meaning is a whole line (CSV row, log entry, sentence) | | You want type conversion (nextInt, nextDouble) for free | You will parse the line yourself (e.g., line.split(",")) | | You want to skip over blank lines and extra spaces automatically | You need to preserve indentation, embedded spaces, or empty lines |
The CSCD 210 typed-file convention is line mode for the type tag (because it is exactly one line) and token mode for the values (because nextInt/nextDouble parse-for-you).
In other languages
- Python:
for line in f:is line mode;f.read().split()is token mode. - C:
fgetsis line mode;fscanf("%d", ...)is token mode. - Bash:
read -r lineis line mode;read word(with default IFS) is token mode.