org.netbeans.spi.lexer
Class TokenFactory<T extends TokenId>

java.lang.Object
  org.netbeans.spi.lexer.TokenFactory<T>

public final class TokenFactory<T extends TokenId>
extends Object

Lexer should delegate all the token instances creation to this class.
It's not allowed to create empty tokens.

Field Summary

static Token

SKIP_TOKEN Deprecated. Use isSkipToken(Token) instead.

Method Summary

Token<T>	createCustomTextToken(T id, CharSequence text, int length, PartType partType) Deprecated. This method is deprecated without replacement - see description how a similar effect can be obtained.
Token<T>	createPropertyToken(T id, int length, TokenPropertyProvider<T> propertyProvider) Create complete token with properties.
Token<T>	createPropertyToken(T id, int length, TokenPropertyProvider<T> propertyProvider, PartType partType) Create token with properties.
Token<T>	createToken(T id) Create token with token length corresponding to the number of characters read from the lexer input.
Token<T>	createToken(T id, int length) Create regular token instance with an explicit length.
Token<T>	createToken(T id, int length, PartType partType) Create regular token instance with an explicit length and part type.
Token<T>	getFlyweightToken(T id, String text) Get flyweight token for the given arguments.
boolean	isSkipToken(Token<T> token) Check whether a token (produced by one of the token creation methods) is a special flyweight token used in cases when there is an active filtering of certain token ids (e.g.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SKIP_TOKEN

public static final Token SKIP_TOKEN

Deprecated. Use isSkipToken(Token) instead.

Token instance that the token creation methods in this class produce if there is an active filtering of certain token ids and the just recognized token-id should be skipped. Normally lexers do not need to check for this except some specific cases in which the isSkipToken(Token) is a better typed alternative to this field.

Method Detail

createToken

public Token<T> createToken(T id)

Create token with token length corresponding to the number of characters read from the lexer input.

See Also:

createToken(TokenId, int)

createToken

public Token<T> createToken(T id,
                            int length)

Create regular token instance with an explicit length.

Parameters:

id - non-null token id recognized by the lexer.

length - >=0 length of the token to be created. The length must not exceed the number of characters read from the lexer input.

Returns:

non-null regular token instance.
SKIP_TOKEN will be returned if tokens for the given token id should be skipped because of token id filter.

createToken

public Token<T> createToken(T id,
                            int length,
                            PartType partType)

Create regular token instance with an explicit length and part type.
This is suitable e.g. for unfinished block comment when a COMMENT token and PartType.START arguments would be used.

Parameters:

id - non-null token id recognized by the lexer.

length - >=0 length of the token to be created. The length must not exceed the number of characters read from the lexer input.

partType - whether this token is complete token or a part of a complete token.

Returns:

non-null regular token instance.
SKIP_TOKEN will be returned if tokens for the given token id should be skipped because of token id filter.

getFlyweightToken

public Token<T> getFlyweightToken(T id,
                                  String text)

Get flyweight token for the given arguments.
Note: The returned token will not be flyweight under certain conditions - see return value description.

Parameters:

id - non-null token id.

text - non-null text that the flyweight token should carry.

Returns:

non-null flyweight token instance.
For performance reasons there is a limit for number of successive flyweight tokens. If this limit would be exceeded a single non-flyweight token gets created instead of flyweight one.
SKIP_TOKEN will be returned if tokens for the given token id should be skipped because of token id filter.

createPropertyToken

public Token<T> createPropertyToken(T id,
                                    int length,
                                    TokenPropertyProvider<T> propertyProvider)

Create complete token with properties.

Parameters:

id - non-null token id.

length - >=0 length of the token to be created. The length must not exceed the number of characters read from the lexer input.

propertyProvider - token property provider or null if there are no extra properties. See TokenPropertyProvider for examples how this parameter may be used.

Returns:

non-null property token instance.
SKIP_TOKEN will be returned if tokens for the given token id should be skipped because of token id filter.

createPropertyToken

public Token<T> createPropertyToken(T id,
                                    int length,
                                    TokenPropertyProvider<T> propertyProvider,
                                    PartType partType)

Create token with properties.

Parameters:

id - non-null token id.

length - >=0 length of the token to be created. The length must not exceed the number of characters read from the lexer input.

propertyProvider - token property provider or null if there are no extra properties. See TokenPropertyProvider for examples how this parameter may be used.

partType - whether this token is complete or just a part of complete token. Null may be passed which implies PartType.COMPLETE.

Returns:

non-null property token instance.
SKIP_TOKEN will be returned if tokens for the given token id should be skipped because of token id filter.

createCustomTextToken

public Token<T> createCustomTextToken(T id,
                                      CharSequence text,
                                      int length,
                                      PartType partType)

Deprecated. This method is deprecated without replacement - see description how a similar effect can be obtained.

Create token with a custom text that possibly differs in length and content from the text represented by the token in the input text.
Note: This method should not be used. It is planned to be removed completely. The custom text tokens no longer save space by not refrencing the original characters (when read e.g. from a Reader).
Having token's text to always match the input's text is more systematic and simplifies the lexer module's design.
Therefore the only benefit of custom text tokens would be if certain tools e.g. parsers would require a different text than the one present naturally in the token. In such case the token should have a property (the key can be e.g. a CharSequence.class) that will return a char sequence with the desired text. If the text is a sub sequence of original token's text the token property provider can even be made flyweight:

 StripFirstAndLastCharTokenPropertyProvider implements TokenPropertyProvider {
     public TokenPropertyProvider INSTANCE = new StripFirstAndLastCharTokenPropertyProvider();
     public Object getValue(Token token, Object key) {
         if (key == CharSequence.class) {
             return token.text().subSequence(1, token.length() - 1);
         }
         return null;
     }
 }
 

Parameters:

id - non-null token id of the token being created.

text - non-null custom text assigned to the token.

length - recognized characters corresponding to the token being created.

partType - should always be null otherwise this method would throw an exception.

isSkipToken

public boolean isSkipToken(Token<T> token)

Check whether a token (produced by one of the token creation methods) is a special flyweight token used in cases when there is an active filtering of certain token ids (e.g. comments and whitespace) and the just recognized token-id should be skipped.

Parameters:

token - non-null token.

Returns:

true if the token is a skip-token.