MathLib.precision returns the maximum number of digits that are possible in a variable value.
eglx.lang
static function precision(value smallint in) returns(int); static function precision(value int in) returns(int); static function precision(value bigint in) returns(int); static function precision(value float in) returns(int); static function precision(value smallfloat in) returns(int); static function precision(value decimal in) returns(int);
After the following code runs, the value of result is 9:
result, myVar INT; result = mathLib.precision(myVar);
In EGL, precision is the total number of digits a variable value, not just the number of decimal places. For example, the precision of an INT value is 9. For floating-point numbers, the precision is the maximum number of digits that the number can represent on the system on which the precisionic is running.
In the case of Java™, EGL uses methods in the Java StrictMath class that are equivalent to the EGL functions. This usage ensures that the runtime behavior is the same for every Java Virtual Machine.
Target | Issue |
---|---|
Java | No issues. |
JavaScript | No issues. |