Class Tokenizer

Represents a generic tokenizer.

Inheritance

Object

Tokenizer

Inherited Members

Object.ToString()

Object.Equals(Object)

Object.Equals(Object, Object)

Object.ReferenceEquals(Object, Object)

Object.GetHashCode()

Object.GetType()

Object.MemberwiseClone()

Namespace: Swan.Parsers

Syntax

public abstract class Tokenizer

Constructors

Tokenizer(String)

Initializes a new instance of the Tokenizer class. This constructor will use the following default operators:

Operator	Precedence
=	1
!=	1
>	2
<	2
>=	2
<=	2
+	3
&	3
-	3
*	4
(backslash)	4
/	4
^	4

Declaration

protected Tokenizer(string input)

Parameters

Type	Name	Description
String	input	The input.

Tokenizer(String, IEnumerable<Operator>)

Initializes a new instance of the Tokenizer class.

Declaration

protected Tokenizer(string input, IEnumerable<Operator> operators)

Parameters

Type	Name	Description
String	input	The input.
IEnumerable<Operator>	operators	The operators to use.

Properties

Tokens

Gets the tokens.

Declaration

public List<Token> Tokens { get; }

Property Value

Type	Description
List<Token>	The tokens.

Methods

EvaluateFunctionOrMember(String, Int32)

Evaluates the function or member.

Declaration

public virtual bool EvaluateFunctionOrMember(string input, int position)

Parameters

Type	Name	Description
String	input	The input.
Int32	position	The position.

Returns

Type	Description
Boolean	true if the input is a valid function or variable, otherwise false.

GetDefaultOperators()

Gets the default operators.

Declaration

public virtual Operator[] GetDefaultOperators()

Returns

Type	Description
Operator[]	An array with the operators to use for the tokenizer.

ResolveFunctionOrMemberType(String)

Resolves the type of the function or member.

Declaration

public abstract TokenType ResolveFunctionOrMemberType(string input)

Parameters

Type	Name	Description
String	input	The input.

Returns

Type	Description
TokenType	The token type.

ShuntingYard(Boolean)

Shunting the yard.

Declaration

public virtual IEnumerable<Token> ShuntingYard(bool includeFunctionStopper = true)

Parameters

Type	Name	Description
Boolean	includeFunctionStopper	if set to true [include function stopper] (Token type Wall).

Returns

Type	Description
IEnumerable<Token>	Enumerable of the token in in.

ValidateInput(String, out Int32)

Validates the input and return the start index for tokenizer.

Declaration

public abstract bool ValidateInput(string input, out int startIndex)

Parameters

Type	Name	Description
String	input	The input.
Int32	startIndex	The start index.

Returns

Type	Description
Boolean	true if the input is valid, otherwise false.

Comments