%************************************************
\chapter{Achieved Results}\label{ch:achieved} % $\mathbb{ZNR}$
%************************************************

\section{Symbolic approach to quantified bit-vectors}
Our research is focused on employing symbolic methods for deciding
satisfiability of quantified bit-vector formulas. In particular, we
have developed a \bdd-based solver Q3B for quantified bit-vector
logic. The solver uses simplifications like early quantification in
order to reduce the size of the intermediate \bdds, tailored \bdd
variable ordering based on dependencies in the input formula, which
can further reduce the size of \bdds, and approximations, which
partially alleviate infeasibility of representing multiplication with
a \bdd. This section summarizes all three of these components, which
were in detail described in our paper published at the SAT 2016
conference~\cite{JS16}.

The benefit of a \bdd-based solver over the quantifier
instantiation-based one is the ability of representing all models of a
universally quantified formula even in cases in which finding a
suitable set of quantifier instantiations would be infeasible. The
following formula, in which all variables have bit-width 32 bits,
shows this difference:
\[
  \varphi \equiv a=16 \cdot b \,+\, 16 \cdot c~\wedge~\forall (x :
  [32]) \, (a \not = 16 \cdot x).
\]

The \bdd-based solver can quickly compute the \bdd for the quantified
subformula $\forall (x : [32]) \, (a \not = 16 \cdot x)$, which shows
that at least one of the last 4 bits of the variable $a$ has to be
$1$, i.e. the value of $a$ is not divisible by $16$. After conjoining
this \bdd to the \bdd for $a=16 \cdot b \,+\, 16 \cdot c$, the formula
is decided unsatisfiable, as the resulting \bdd represents the
constant $0$ function. On the other hand, if one considers only
quantifier instantiations by the subterm of the input formula,
exponentially many quantifier instances have to be added to the
formula to show its unsatisfiability, as there is no subterm of
$\varphi$ that can be instantiated as $x$ to yield an unsatisfiable
quantifier-free formula.

\subsection{Simplifications}

Simplifications used in Q3B aim for reducing the size of the
intermediate \bdds. This can be achieved, besides partially canonizing
the subterms, by reducing scopes of quantifiers occurring in the
formula. One such simplification is distributivity of universal
quantification over conjunction:
\[
  \forall x. \, (\varphi ~\wedge~ \psi)~~\leadsto~~(\forall x . \,
  \varphi) ~\wedge~ (\forall x . \, \psi).
\]
After such transformation, the \bdd for $\varphi \wedge \psi$ does not
have to be computed and the conjunction of the \bdds is computed only
after elimination of the universal quantifier, which usually reduces
the size of the formula. Dually, the existential quantification can be
distributed over disjunctions.

Moreover, universal quantification can distribute over disjunctions in
cases where the variable bound by the quantifier does not occur in one
of the disjuncts. This leads to the following rule and its dual
version, which is known as \emph{miniscoping}~\cite{Har09}:
\[
  \forall x. \, (\varphi[x] ~\vee~ \psi)~~\leadsto~~(\forall x . \,
  \varphi[x]) ~\vee~ \psi,
\]
where $x$ does not occur freely in $\psi$. Note that the scope of the
quantifier is again reduced, which can lead to smaller intermediate
\bdds.

Q3B also uses \emph{destructive equality resolution} (\der) rule,
which was proposed for quantified bit-vector formulas by Wintersteiger
et al:
\[
  \forall x. \, (x \not = t ~\vee~ \varphi[x])~~\leadsto~~\varphi[t],
\]
where $t$ is an arbitrary term that does not contain $x$. As the
\der rule eliminates the quantified variable, it in many cases also
reduces the size of the \bdds.

Unlike Z3, for which the \der rule was proposed, our solver does not
perform Skolemization before solving. Therefore we also need a dual
version of the \der rule, which we have called \emph{constructive
  equality resolution}:
\[
  \exists x. \, (x = t ~\wedge~ \varphi[x])~~\leadsto~~\varphi[t],
\]
where $t$ is an arbitrary term that does not contain $x$.


\subsection{Variable ordering}
Ordering of \bdd variables is crucial for efficiency. The size of a
\bdd can differ exponentially when choosing a different
ordering. Therefore, besides well known methods of dynamic variable
reordering during the computation, Q3B computes an initial variable
ordering, which is based on the syntactic dependencies among the
variables in the formula.

We have implemented and compared three different initial orderings,
which we denote $\leq_1, \leq_2, \leq_3$.
\begin{itemize}
\item In $\leq_1$, all bit variables are ordered according to their
  significance (from the least to the most significant) and variables
  with the same significance are ordered by the order of the first
  occurrences of the corresponding bit-vector variables in the
  formula. In this ordering, bit variables of the same significance
  are close to each other.
\item In $\leq_2$, bit variables are ordered by the order of the first
  occurrences of the corresponding bit-vector variables in the formula
  and bit variables corresponding to the same bit-vector variable are
  ordered according to their significance (from the least to the most
  significant). In this ordering, bit variables corresponding to the
  same bit-vector variable are close to each other.
\item In $\leq_3$, we first identify syntactically dependent
  variables. Two variables are \emph{syntactically dependent} if they
  occur in the same atomic subformula. Then the set of variables is
  decomposed to equivalence classes modulo the transitive closure of
  the syntactic dependence relation and each of these classes is
  ordered according to $\leq_2$. Therefore, in this ordering, only
  potentially dependent bit variables of the same significance are
  close to each other.
\end{itemize}
From these three orderings, $\leq_3$ performs best on the set of our
benchmarks even if the dynamic variable reordering by
\emph{sifting}~\cite{Rud93} is used.

\subsection{Approximations}
The size of the \bdd for multiplication is exponential regardless the
variable ordering~\cite{Bry91}. Therefore, to be able to decide
formulas with multiplication or complex arithmetic, Q3B uses
approximations of the input formula, which allow deciding a
satisfiability of the input formula by solving a simpler formula
instead. In the context of \smt solving, an underapproximation of a
formula is a formula that logically entails the input formula and an
overapproximation of an input formula is a formula logically entailed
by the input formula. It can be easily seen that satisfiability of an
underapproximation implies satisfiability of the input formula and
unsatisfiability of an overapproximation implies unsatisfiability of
the input formula.

Q3B employs approximations similar to those used by the \smt solver
\uclid~\cite{BKOSSB07}. In \uclid, the quantifier-free input formula
is underapproximated by representing each bit-vector variable by the
smaller number of bit variables than its original bit-width. The
number of bits by which the bit-variable is represented is called the
\emph{effective bit-width}. For reducing the effective bit-width,
\uclid offers multiple ways to represent a variable by a smaller
number of bits: \emph{zero extension}, \emph{one-extension}, and
\emph{sign-extension}. In all three, the effective bit-width is used
to represent the least significant bits of the bit-vector variable and
all other bits are set to 0, 1, or the value of the most significant
effectively represented bit, respectively. To these extensions, we
have proposed their right variants, which use effective-bit width to
represent the most significant bits. Figure \ref{fig:extensions}
illustrates left and right variants of zero-extension and
sign-extension.

\begin{figure}[tb]
      \begin{tabularx}{\textwidth}{c X c}
        \newcolumntype{C}{>{\centering\arraybackslash}p{2ex}}
        \begin{tabular}{ | C | C | C | C | C | C | }
          \hline
          0 & 0 & 0 & $a_2$ & $a_1$ & $a_0$ \\
          \hline
        \end{tabular}
            &&
               \newcolumntype{C}{>{\centering\arraybackslash}p{2ex}}
               \begin{tabular}{ | C | C | C | C | C | C | }
                 \hline
                 $a_2$ & $a_2$ & $a_2$ & $a_2$ & $a_1$ & $a_0$ \\
                 \hline
               \end{tabular}
        \\ zero-extension && sign-extension
        \\ [1ex]
        \newcolumntype{C}{>{\centering\arraybackslash}p{2ex}}
        \begin{tabular}{ | C | C | C | C | C | C | }
          \hline
          $a_5$ & $a_4$ & $a_3$ & 0 & 0 & 0 \\
          \hline
        \end{tabular}
            &&
               \newcolumntype{C}{>{\centering\arraybackslash}p{2ex}}
               \begin{tabular}{ | C | C | C | C | C | C | }
                 \hline
                 $a_5$ & $a_4$ & $a_3$ & $a_3$ & $a_3$ & $a_3$ \\
                 \hline
               \end{tabular}
        \\
        right zero-extension && right sign-extension
      \end{tabularx}
      \caption{Reductions using zero-extension and sign-extension of a
        6-bit variable $a=a_5a_4a_3a_2a_1a_0$ to $3$ effective bits.}
    \label{fig:extensions}
\end{figure}

In contrast to \uclid, we can use the same approach to perform
overapproximations, because we work with the quantified formulas. For
formulas in the negation normal form, underapproximations can be
achieved by reducing effective bit-widths of existentially quantified
variables and overapproximations can be achieved by reducing effective
bit-widths of universally quantified variables.

In Q3B, approximations are used in a portfolio style. The solver
without approximations is run in parallel with the solver which
employs underapproximations and the solver which employs
overapproximations. Our experimental evaluation has shown that this
set up can help to decide more formulas, especially formulas
containing multiplication.

\subsection{Experimental evaluation}

Our experimental evaluation has shown that a \bdd-based \smt solver
can decide more formulas than CVC4 and Z3, when the set of quantified
bit-vector formulas in the SMT-LIB benchmark library~\cite{BST10} is
considered. Additionally, we have gathered quantified formulas
produced by the semi-symbolic model checker \SymDivine~\cite{BHB14},
which uses quantified formulas to decide the equality of two symbolic
states. On this set of benchmarks, our solver Q3B also has better
performance than \smt solvers based on the quantifier instantiation
and bit-blasting. Table \ref{tbl:results} shows the numbers of
formulas solved by each solver and Figure \ref{fig:quantilePlots}
presents the quantile plot of CPU times needed to solve these
formulas.

All experiments were performed on a Debian machine with two six-core
Intel Xeon E5-2620 2.00GHz processors and 128 GB of RAM. Each
benchmark run was limited to use 3 processor cores, 4 GB of RAM and 20
minutes of CPU time (if not stated otherwise).

\begin{table}
\checkoddpage
\edef\side{\ifoddpage l\else r\fi}
\makebox[\textwidth][\side]{
\begin{minipage}[bt]{\fullwidth}
  \begin{center}
    \setlength{\tabcolsep}{0.5em}
    \begin{tabularx}{\textwidth}{l r r r r r X r r r r}
      \toprule
      & ~ & \multicolumn{4}{c}{SMT-LIB} & ~ & \multicolumn{4}{c}{\SymDivine} \\
      \cmidrule(l){3-6}
      \cmidrule(l){8-11}
      & & sat & unsat & unknown & timeout & & sat & unsat & unknown & timeout \\
      \midrule
      CVC4 & & $29$ & $55$ & $32$ & $75$ & & $1\,124$ & $3\,845$ & $2$ & $490$  \\
      Z3 & & $71$ & $93$ & $5$ & $22$ & & $1\,135$ & $4\,162$ & $22$ & $142$ \\
      Q3B & & $\mathbold{94}$ & $\mathbold{94}$ & $0$ & $3$ & & $\mathbold{1\,137}$ & $\mathbold{4\,202}$ & $0$ & $122$  \\
      \bottomrule
    \end{tabularx}
  \end{center}
  \caption{For each benchmark set and each solver, the table provides the
    numbers of formulas decided as satisfiable (\emph{sat}),
    unsatisfiable (\emph{unsat}), or undecided with the result unknown or because of
    an error (\emph{unknown}), or a \emph{timeout}.}
  \label{tbl:results}
\end{minipage}}
\end{table}

\begin{figure}
\checkoddpage
\edef\side{\ifoddpage l\else r\fi}
\makebox[\textwidth][\side]{
\begin{minipage}[bt]{\fullwidth}
  \begin{subfigure}%{0.3\textwidth}
    \centering
    \includegraphics{gfx/comparisonSmtlib.pdf}
  \end{subfigure}
  \hfill
  \hskip-.5\textwidth
  \begin{subfigure}%{0.57\textwidth}
    \centering
    \includegraphics{gfx/comparisonSymdivine.pdf}
  \end{subfigure}
  \caption{Quantile plot of the number of benchmarks which CVC4, Q3B,
    and Z3 solved compared by the CPU time.}
  \label{fig:quantilePlots}
\end{minipage}}
\end{figure}

\begin{table}
  \checkoddpage
  \edef\side{\ifoddpage l\else r\fi}
  \makebox[\textwidth][\side]{
    \begin{minipage}[t]{\fullwidth}
    \setlength{\tabcolsep}{0.5em}
    \begin{tabularx}{\textwidth}{l r r r X r r r}
      \toprule
      & ~ & \multicolumn{2}{c}{Known status} & & \multicolumn{3}{c}{Unknown status}  \\
      \cmidrule(l){3-4} \cmidrule(l){6-8} & & \# solved & avg. CPU
      time & &\# solved & avg. CPU time &
      avg. WALL time \\
      \midrule
      Boolector & & $85$ & $1.635$ & & $89$ & $\mathbold{11\,431}$ & $11\,422$ \\
      CVC4 & & $85$ & $1.576$ & & $56$ & $29\,464$ & $29\,453$ \\
      Q3B & & $85$ & $\mathbold{0.138}$ & & $\mathbold{99}$ & $12\,111$ & $\mathbold{4\,059}$ \\
      Z3 & & $85$ & $0.339$ & & $78$ & $16\,721$ & $16\,713$ \\
      \bottomrule
    \end{tabularx}
    \caption{Official results of quantified bit-vector category of
      \smt competition 2016 divided into the benchmarks with known
      status and benchmarks with a previously unknown status. All
      times are in seconds.}
    \label{tbl:smtcomp}
  \end{minipage}}
\end{table}

Recently, these results were confirmed by the \smt competition
2016\footnote{\url{http://smtcomp.sourceforge.net/2016/}} -- our
solver is the winner of the quantified-bit vectors category. Table
\ref{tbl:smtcomp} shows official results for this category. The
benchmarks are divided into two groups, with \emph{known} and
\emph{unknown} status. A benchmark has a known status if at least two
solvers in the previous year of the competition agreed whether the
benchmark is satisfiable or unsatisfiable. The results show that Q3B
can solve as many known benchmarks as the other solvers, but solves
them in the shortest time. Moreover, Q3B can solve more benchmarks
with unknown status than any of the other solvers.

\section{Other areas of research}
Outside the field of \smt solving, my research interests also include
software verification. In this field, I have co-authored the following
paper on the tool Symbiotic, which combines instrumentation, slicing
and, symbolic execution to allow verification of a real-world
code~\cite{CJSSV16}.


\section{Published papers}


\begin{itemize}
\item \textsc{Jonáš}, M. and J. \textsc{Strejček}. "\emph{Solving Quantified Bit-Vector
    Formulas Using Binary Decision Diagrams.}" In: Theory and
  Applications of Satisfiability Testing -- SAT 2016 -- 19th
  International Conference, Bordeaux, France, July 5-8, 2016,
  Proceedings. 2016, pp. 267-283~\cite{JS16}
  \smallskip

  I am the main author of the text of the paper. I have also
  implemented the \smt solver and conducted all the experiments.
\item \textsc{Chalupa} M., M. \textsc{Jonáš}, J. \textsc{Slabý}, J. \textsc{Strejček}, and
  M. \textsc{Vitovská}. "\emph{Symbiotic 3: New Slicer and Error-Witness
    Generation -- (Competition Contribution).}" In: Tools and
  Algorithms for the Construction and Analysis of Systems -- 22nd
  International Conference, TACAS 2016, Held as Part of the ETAPS
  2016, Eindhoven, The Netherlands, April 2-8, 2016, Proceedings,
  pp. 946-949~\cite{CJSSV16}

  \smallskip

  I wrote parts of the paper and prepared the environment that was
  used to run experiments with the implemented tool Symbiotic.
\end{itemize}



%*****************************************
%*****************************************
%*****************************************
%*****************************************
%*****************************************

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../ClassicThesis"
%%% End: