Deterministic and Nondeterministic Functions

A deterministic function is a function that always evaluates to the same result if it is invoked multiple times with the same arguments during the evaluation of a snapshot. In general, all functions defined by XQuery are deterministic. However, a module developer might develop a function (in a host language) which does not guarantee to be deterministic. For example, a function generating random numbers is clearly not deterministic. Knowing about this property is crucial for the query optimizer to retain the semantics (e.g. caching would lead to incorrect results).For this reason, Zorba provides two annotations (nondeterministic and deterministic) that allow a developer to declare whether a function is deterministic or not.As an example, the following snippet declares a nondeterministic random function.

declare namespace an = "http://zorba.io/annotations";

declare %an:nondeterministic function random:random() as xs:integer external;

If not specified otherwise, all functions are being treated as deterministic. If some function invokes a nondeterministic function, the invoking function itself must also be declared as nondeterministc, otherwise zerr::ZXQP0040 is raised.

Sequential Functions

In XQuery 3.0, an expression never has any side effects other than constructing new nodes (not even updating expressions, which merely produce a pending update list). Zorba Scripting programs may have side effects. Side effects can be made with statements. An expression is sequential if it contains statements that have side effects (updating some XML, assigning a variable in scope outside of the expression) or that affect the control flow (break, continue, exit returning), or if it invokes a sequential function. Analogously to nondeterministic functions, the compiler needs to be able to infer if an expression is sequential. In order to do so, Zorba provides the an:sequential and an:nonsequential annotations. The an:sequential annotation must be used to declare (external) functions that have side effects. If non of these annotations is present, the default for a function is to be annotated as an:nonsequential. If a function invokes a sequential function but is itself not declared sequential (i.e. no annotation or an:nonsequential), an error is raised (zerr:XSST0004).For example, the function of the http-client module is declared as sequential because invoking it might cause a side effect (e.g. a payment with paypal).

declare %an:sequential function http:post(
  $href as xs:string,
  $body as item(),
  $content-type as xs:string) as item()+ { ... }

Assignable and Nonassignale Variables

In plain XQuery without scripting, global variables (i.e. those declared in the prolog of a module) can not be assigned a value. In scripting (i.e. if the scripting feature is enabled), however, values can be assigned to a variable.In order to allow the developer to prevent variable assignments (even with scripting), Zorba provides the an:nonassignable annotation.For example, preventing a variable from being assigned in scripting mode could be done as follows:

declare namespace an = "http://zorba.io/annotations";

declare %an:nonassignable variable $var := 3;

If a value is assigned to a nonassignable variable, the error XSST0007 in the Zorba error namespace is raised.

Variadic Functions

A function annotated with the an:variadic annotation is a function of indefinite arity, i.e. one that accepts a variable number of arguments.For example, the function to create an unordered map with an arbitrary number of key types (see module http://zorba.io/modules/store/data-structures/unordered-map) is declared as follow:

declare %an:variadic %an:sequential function map:create(
  $name as xs:QName,
  $key-type as xs:QName) as empty-sequence() external;

Streamable Strings

A function annotated with the streamable annotation is a function that may return an xs:string item whose content is streamed. Such a string is called a streamable string. They have the advantage that their contents does not need to be materialized in memory. If a function consuming such a string is able to process the string in a streaming fashion, this allows for processing of strings with a virtually infinite length.For example, the following XQuery code, reads a file from disk and returns the contents to the standard output of the calling process. Because the file:read-text function is annotated using the streamable annotation, the file does not have to be materialized in memory.

import module namespace file = "http://expath.org/ns/file";

file:read-text("big_file.txt")

However, the disadvantage is that a streamable string can only be consumed exactly once. If a streamable string is consumed more than once, an error is raised. In order to enable multiple consumers of a streamable string, the materialize function of the string module (http://zorba.io/modules/string) should be used to materialize the entire contents in an (regular) xs:string item.

import module namespace file = "http://expath.org/ns/file";
import module namespace string = "http://zorba.io/modules/string";

let $x := string:materialize(file:read-text("myfile.txt")))
return ($x, $x)

In this example, the function returns a streamable string whose contents is used twice in the query. In order to be able to use the value twice, the string:materialize function must be used to materialize the entire contents of the file myfile.txt in memory. Otherwise, the error zerr:ZSTR0055 is raised.

Caching Results of Functions

Caching of function results may improve performance when computationally expensive functions are invoked multiple times with the same arguments.If the optimization level is O1 or higher, Zorba automatically caches results of recursive, deterministic, and non-sequential functions whose parameter and return types are subtypes of xs:anyAtomicType. Specifically, if such a function is called more than once with the same arguments, the result of the first call will be cached and subsequent calls will return the cached value without re-evaluating the function.For example, in the following recursive function computing a fibonacci number, each result is automatically cached and, hence, dramatically improves performance.

declare function local:fib($n as xs:integer) as xs:integer
{
  if ($n eq 0) then 0 
  else if ($n eq 1) then 1 
  else local:fib($n - 1) + local:fib($n - 2)
};

local:fib(100)

Specifically, this optimization reduces the complexity of the function from O(1.6^n) to O(n).In order to explicitly disable function caching, the user can specify the an:no-cache annotation.In addition, the user can use the an:cache annotation to cache the results of functions other than the ones that are automatically cached. However, this will only work if the function is not updating and its parameter and return types are subtypes of xs:anyAtomicType; otherwise, Zorba will raise a warning (zwarn:ZWST0005) and simply ignore the an:cache annotation.Please note, that explicitly enforcing caching for sequential or nondeterministic functions might not give the intended result. In such cases, Zorba will raise a warning (zwarn:ZWST0006) but obey the an:cache annotation.

Avoiding Node Copying in Node Constructors.

According to the XQuery specification, any nodes (and their subtrees) returned by an enclosed expression appearing inside a node constructor must be copied, and it is the copied subtrees that are then connected to the newly constructed node. However, such copying is not always necessary. If no copying is done, then some nodes will be "shared" by multiple trees. XML trees that do not contain any shared nodes will be referred to as "stand-alone trees".Whether node copying is really necessary or not depends on the kind of operations that are going to be performed on the nodes and their associated trees after the node construction. For example, copying is necessary if any operations involving node identity, document ordering, or reverse axis traversals are to be performed on the newly constructed tree. Copying is also necessary if the relevant nodes may be involved in a later update operation. The Zorba optimizer is able to analyze the query expressions and determine automatically whether copying is required or not. However, the optimizer needs some help from the user in the case of external functions, because it cannot look into the operations of such functions. For this, Zorba provides two annotations that users may annotate their external functions with in order to help the optimizer determine whether to copy nodes or not.

• an:must-copy-input-nodes. This annotation has an associated value, which is a list of integer literals. The meaning of the annotation is the following: If a number N appears in the value, then the function requires that any nodes that are bound to the N-th parameter when the function is called, must belong to standalone trees (because the function is going to perform on such nodes an operation that requires standalone trees). As a result of this information, the optimizer makes sure that node copying is done during any node constructor whose result (or any subtree of it) may be given as input to such an external function.
• an:propagates-input-nodes. This annotation has an associated value, which is a list of integer literals. The meaning of the annotation is the following: If a number N appears in the value, then the function may return to its caller a node n that is bound to the N-th parameter, or a node in the subtree of n. This annotation helps the optimizer determine whether nodes consumed by an expression are coming from a node constructor, so that if the expression requires stand-alone trees, then the optimizer will make sure that the relevant node constructor does copy nodes.

The absence of any of the above annotations from an external function declaration is treated as if the annotation was present with a value listing all the numbers between 1 and M, where M is the function arity.

Annotations on Collections and Indexes

The Data Definition Facility uses annotations to assign properties to collections and indexes. For example, annotations can be used to specify that a collection is unordered or that an index is a value equality index.The semantics of each of the annotations is given in the documentation about the XQuery Data Definition Facility. However, for completeness, we also list all of the annotations here.

• an:mutable, an:queue, an:append-only, an:const Determines whether a collection is mutable, queue (FIFO), append-only, or const (not mutable). The default annotation is mutable.
• an:ordered, an:unordered Annotations to specify whether the order of nodes in a collection must be preserved. The default is ordered.
• an:read-only-nodes, an:mutable-nodes If a collection is annotated with read-only-nodes, the nodes in the collection can not be modified using the XQuery Update Facility. The default is an:mutable-nodes.
• an:unique, an:nonunique Unique indexes make sure that the relationship between index keys and values is one-to-one. The default is an:nonunique.
• an:value-equality, an:value-range, an:general-range, an:general-equality Determine whether the index is a value or general equality or value range index, respectively. The default is an:value-equality.
• an:automatic, an:manual If an index is declared as automatic, Zorba guarantees that the index is maintained automatically. The default is an:manual.