• A string can be divided into tokens using Java’s StringTokenizer class.
•  Internally, a StringTokenizer object keeps track of where it is in the string that has to be tokenized. 
• Some procedures move this place ahead of the currently processed characters.
•  By extracting a substring from the string that was used to generate the StringTokenizer object, a token is returned.
•  It offers the initial stage of the parsing procedure, often known as the lexer or scanner. 
• The String Tokenizer class enables applications to tokenize strings. The Enumeration interface is implemented by it. 
• Data parsing is done using this class. 
• We must specify an input string and a string with delimiters  to use the String Tokenizer class 
• The characters that divide tokens are known as delimiters.
• The delimiter string’s characters are all accepted as delimiters. Whitespaces, new lines, spaces, and tabs are the default delimiters.

Constructors of StringToken: 

Let us consider ‘str’ as the string to be tokenized

1. StringTokenizer(String str): Delimiters that are typically used include newline, space, tab, carriage return, and form feed.
2. StringTokenizer(String str, String delim):  To tokenize the supplied string, a collection of delimiters called delim is used.
3. StringTokenizer(String str, String delim, boolean flag):The meaning of the first two parameters is the same, and the flag has the following function.

3.1: If the flag is false, delimiter characters serve to separate tokens

Example:

Input : if string --> "hello coderz" and Delimiter is " ", then 
Output:  tokens are "hello" and "coderz".

3.2: If the flag is true, delimiter characters are considered to be tokens.

Example:

Input : String --> is "hello coderz"and Delimiter is " ", then 
Output: Tokens --> "hello", " " and "coderz".

Methods of the StringTokenizer Class:

Methods	Description
boolean hasMoreTokens()	It determines if more tokens are accessible.
String nextToken()	It returns the next token from the StringTokenizer object.
String nextToken(String delim)	Based on the delimiter, it returns the subsequent token.
boolean hasMoreElements()	It is the same as hasMoreTokens() method.
Object nextElement()	It is the same as nextToken() but its return type is Object.
int countTokens()	It returns the total number of tokens.

Example to illustrate the use of StringTokenizer class:

// Java Program to Illustrate StringTokenizer Class

// Importing required classes
import java.util.*;

// Main class
public class GFG {

	// Main driver method
	public static void main(String args[])
	{

		// Constructor 1
		System.out.println("Using Constructor 1 - ");

		// Creating object of class inside main() method
		StringTokenizer st1 = new StringTokenizer(
			"Hello Coderz, keep coding - keep cheering", " ");

		// Condition holds true till there is single token
		// remaining using hasMoreTokens() method
		while (st1.hasMoreTokens())

			// Getting next tokens
			System.out.println(st1.nextToken());

		// Constructor 2
		System.out.println("Using Constructor 2 - ");

		// Again creating object of class inside main()
		// method
		StringTokenizer st2 = new StringTokenizer(
			"JAVA : Code : String", " :");

		// If tokens are present
		while (st2.hasMoreTokens())

			// Print all tokens
			System.out.println(st2.nextToken());

		// Constructor 3
		System.out.println("Using Constructor 3 - ");

		// Again creating object of class inside main()
		// method
		StringTokenizer st3 = new StringTokenizer(
			"JAVA : Code : String", " :", true);

		while (st3.hasMoreTokens())
			System.out.println(st3.nextToken());
	}
}

Output:

Using Constructor 1 - 
Hello
Coderz,
keep
coding
-
keep
cheering
Using Constructor 2 - 
JAVA
Code
String
Using Constructor 3 - 
JAVA:
 
Code
 :
 
String

Note: also read about the StringBuilder class in Java

Follow Me

If you like my post, please follow me to read my latest post on programming and technology.

https://www.instagram.com/coderz.py/

https://www.facebook.com/coderz.py