CSCD210

Line vs token input

Skillcscd210-line-vs-token-inputTextbookBJP Ch 6

What nextLine() actually reads

Student can predict what sc.nextLine() returns given the file's current cursor position, identify that the line terminator is consumed but not part of the return value, and recognize the three cases: non-empty line, empty line, partial last line: by what each returns.

Scanner.nextLine() reads characters from the current cursor position up to and including the next line terminator, then returns everything before the terminator as a String. The terminator is consumed but not returned. The cursor lands one character past it.

//  Suppose the file contains:  Alice\nBob\nCarol\n
final Scanner sc = new Scanner(new File("names.txt"));

String first  = sc.nextLine();   //  "Alice" : cursor now between \n and 'B'
String second = sc.nextLine();   //  "Bob"
String third  = sc.nextLine();   //  "Carol"

Three properties of nextLine() worth committing to memory:

1. The newline is consumed but not returned. The returned String has no \n at the end. Code that appends "\n" for printing is doing the right thing; code that strips a trailing newline from the return is acting on a newline that is not there. 2. Empty lines return empty strings. Two consecutive \n characters mean "an empty line." nextLine() returns "" and the cursor moves past the second newline. Code that filters out blank lines must compare against "" (or use String.isBlank() for whitespace-only lines). 3. The cursor's position before the call decides what is returned. If a previous nextInt() or next() left the cursor mid-line, nextLine() reads from that mid-line position to the next newline. The trap that follows a token-mode read with nextLine covers this in detail, and it is the single most common Scanner bug in CS1.

Line terminator details

Java's Scanner recognizes the three common line endings: \n (Unix), \r\n (Windows), \r alone (old Mac). All three are consumed and produce the same return. A file edited on Windows and read on Linux behaves the same way; the Scanner documentation calls this "line terminator portability."

The trade-off: the String returned never carries the original line ending. A program that needs to preserve "this file was Windows-style" must read the file with Files.readString or similar instead.

When the last line lacks a terminator

Some text files end without a final \n (Windows Notepad sometimes does this; so do programs that fail mid-write). The behavior is:

  • nextLine() reads the partial last line and returns it.
  • The cursor lands at end-of-stream.
  • hasNextLine() returns false after the read.

The lack of a trailing newline does not lose data. The last line is still returned by the last nextLine() call.

In other languages

  • Python: for line in f: includes the trailing \n; line.rstrip() is the typical fix. Java strips it for you.
  • C: fgets(buf, sizeof buf, fp) includes the \n if the buffer was long enough; strcspn or manual trim removes it.
  • JavaScript (Node): readline.Interface .on('line', ...) strips the line terminator like Java does.

next() versus nextInt(): token mode

Student can choose between next(), nextInt(), nextDouble() for reading from a Scanner based on the expected token type, predict the exception thrown when the token cannot be parsed (InputMismatchException), and recognize the cursor-position rule that produces the trailing-whitespace trap.

Scanner has two modes for reading input: line mode and token mode. Token-mode methods read a single whitespace-delimited token and return it. They are the right tool when the file's structure is "values separated by whitespace": spaces, tabs, newlines, any combination.

//  Suppose the file contains:  42 3.14 hello\n  7 2.71 world\n

final Scanner sc = new Scanner(new File("data.txt"));

int    a = sc.nextInt();      //  42
double b = sc.nextDouble();   //  3.14
String c = sc.next();         //  "hello"
int    d = sc.nextInt();      //  7

Three properties of the token-mode methods that distinguish them from nextLine():

1. Whitespace is the delimiter, not the data. Any run of spaces, tabs, and newlines between tokens is skipped automatically. The file 42\n\n\n3.14 reads as two tokens; 42 3.14 reads as two tokens; both behave identically. Reading the same two files with nextLine() would have produced very different output. 2. nextInt() returns the parsed int, not the string. The method parses the token as a base-10 integer. If the token cannot be parsed (e.g., "abc" or "3.14"), InputMismatchException is thrown. Same for nextDouble(), nextLong(), nextBoolean(). 3. **The cursor stops immediately after the last character of the token.** This is the same cursor-position rule that produces the nextInt-then-nextLine trap. After sc.nextInt() reads 42, the cursor sits between 2 and the next character: typically a space or newline that the next token-mode call will skip past but nextLine() would catch as an empty line.

next() reads a string token

sc.next() reads one whitespace-delimited token and returns it as a String, no parsing involved. It is the token-mode analog of nextLine() for line mode:

//  File: alice 30 bob 25 carol 19

while (sc.hasNext()) {
    String name = sc.next();      //  one name token
    int age = sc.nextInt();       //  one int token
    System.out.println(name + " is " + age);
}

Three token-mode reads (next, nextInt, next, nextInt, ...) walk through the alternating name age name age pattern without any line-terminator gymnastics.

The pairing rule

A file format is either line-oriented (one record per line; the record may have internal structure) or token-oriented (whitespace separates every value). Reading line-oriented data with token-mode methods works only when the within-line structure is single-token-per-line: in which case the file is also token-oriented, and either mode is fine. Reading truly line-oriented data (e.g., the line itself is a CSV row that needs to be parsed) requires nextLine() followed by String.split(",").

CSCD 210 Lab 11 (Typed File Stats) uses the token mode for the value lines (every line is one value) and nextLine() for the type tag (which is one whole line by itself, and is read first so no whitespace mixing can occur).

hasNextInt() and friends

The has-test versions exist for each typed read:

while (sc.hasNextInt()) {
    int v = sc.nextInt();
    // ...
}

hasNextInt() looks at the next token without consuming it and returns true only if the token parses as an int. This is the pattern for "read integers until we see something that is not one." Reading until the end of the file covers the loop shapes, and this is the typed variant.

In other languages

  • C: fscanf("%d", &n) is the analog of nextInt(); %s for next(). Same trap with the trailing newline.
  • Python: no built-in token mode; the idiom is line.split() to get a list of strings, then int(...) per token to parse.
  • C++: cin >> n; is the analog; same skip-whitespace, same trailing-newline issue.

The nextLine after nextInt trap

Student can predict that a nextLine call immediately after a nextInt (or any token-mode read) returns the empty string, and apply one of the two standard fixes (discard newline with an extra nextLine, or stay in token mode).

The single most common file-I/O bug in CS1 looks like this:

int n = sc.nextInt();
String name = sc.nextLine();   // students expect: the next line
                               // actually returns: the empty string ""

The diagnosis: nextInt() reads the digits of n and stops immediately after the last digit. The newline character that ended the line is still in the buffer. nextLine() then reads from the current position up to the next newline, which is right there. The return is the empty string, and n's line is effectively "consumed."

This is not a bug in Scanner; it is the documented behavior. The two methods have different rules about where they leave the cursor, and those rules collide.

Reading the buffer character by character

Imagine the file as a stream and a ^ marking the cursor:

^42\nJessica\n

After sc.nextInt(), the cursor sits between the 2 and the \n:

42\nJessica\n
   ^

Now sc.nextLine() runs. Its rule is "consume up to and including the next newline; return what came before." The next newline is the very next character. So the call returns "" and advances the cursor:

42\nJessica\n
    ^

A second sc.nextLine() returns "Jessica".

The two standard fixes

Fix A: flush the newline. After every token-mode read on a line of its own, call sc.nextLine() once and throw the result away:

int n = sc.nextInt();
sc.nextLine();            // discard the newline that nextInt left behind
String name = sc.nextLine();

Fix B: stay in token mode the whole time. If the file's structure permits it, use next() (single token) or nextInt()/nextDouble() (typed token) for every read. Token-mode reads skip whitespace including newlines, so the trap never appears:

int n = sc.nextInt();
String name = sc.next();   // single token; no trap

The CSCD 210 typed-file convention is fix B for the value lines (every value is one token) and Fix A's avoid the mix variant for the type tag: the tag is read with nextLine() before any token-mode reads happen, so there is no trailing whitespace from a prior call.

In other languages

  • Python: the line-vs-token distinction does not exist in the standard library; input() and iterating over a file always return whole lines, and .split() does the tokenizing.
  • C: fscanf("%d", &n) leaves the newline in the buffer the same way nextInt does; fgets reads a line including the newline. The same trap exists.
  • C++: cin >> n; getline(cin, name); has the identical trap with the identical fix (cin.ignore()).