Lecture 10-1 - Their Bark is Worse Than Their Byte - Part II

To motivate our next data structure (a tree!) let’s consider one of the nice tools we had in the past: Binary Search.

Question

What was the chief benefit of binary search compared to the sort of “naive” or brute force approach?

Answer

Binary search allowed us to find items in some sorted list in O(log(n)) time, compared to a linear search which would be a O(n) operation.

Question

What part of using binary search is a somewhat large assumption — something we may not always be able to take for granted?

Answer

That the list is sorted!

Remark

As it turns out, taking any arbitrary list and then sorting it is a nontrivial task — the best sorting algorithms (which we’ll examine later) have a typical runtime complexity of O(n*log(n)).

So, here are the facts:

• Lists are expensive to sort.
• Unsorted lists incur a search performance of O(n).
• However, performing binary search on a sorted list has performance O(log(n)), which is better than O(n).

Let’s see if we can think about a new data structure that can address the above!

Binary Search Trees - Basics

Example

Can you think of a way to use a tree as a natural data structure for the binary search operation? (hint: think binary trees where each node’s children mean something special)?

Have you thought about it?

Don’t lie to me I’ll know!

Anyways, I guess it was something of a giveaway that I titled this section “binary search tree,” that’s one hell of a hint.

Definition

A binary search tree (BST) is a binary tree with the guarantee that data in the left subtree (i.e., the left child and downward) of each node is “less” than the parent’s, and the data in the right subtree is “greater” than the parent’s.

Toolkit

Because of these extra properties (whose utility will soon be revealed), we typically use Binary Search Trees (the data structure) to model Sets (an Abstract Data Type). With sets, and in BSTs, all stored keys (the identifiers of any stored item) are unique (no duplicates).

Here’s an example binary search tree:

Notice that for each node, its left child node is less than it while its right child is greater.

Binary Search Trees - Operations

Now, let’s consider some operations on a binary search tree:

Example

Insert 2 more values into the above tree and explain how you did so: add 7, 24.

Question

Insertion: Explain how to insert a value into a BST; what is the asymptotic runtime complexity of this operation?

Answer

Starting at the root, ask if your insertion is less than or greater than the current node, and follow the path that results until the first open spot. Insertion into a BST again has the complexity O(log(n)) (assuming your tree is “balanced”, which we’ll talk about later).

Question

Binary Search: Explain how binary search works on a BST; what is the asymptotic runtime complexity of this operation?

Answer

Starting at the root, ask if your query is less than or greater than the current node, and follow the path that results. Binary search on a BST again has the complexity O(log(n)) (assuming your tree is “balanced”, which we’ll talk about later).

Question

Sorting: Explain how you may use a BST to easily return a sorted representation of your data. What is the asymptotic runtime complexity of this operation? (hint: think about traversal methods).

Answer

Simply perform an inorder traversal, which we know takes O(n). How elegant!

Binary Search Trees - Implementation

Well, we’ve seen them in action… now let’s see how to build a BST!

Unlike the preceding tree implementations (which were not governed by any rules regarding the semantics of parent-children relationships), we might now create a public BST class instead of a BSTNode, since it would be problematic for a user to modify our BST structure.

We’ll start by scaffolding the BSTNode and then work on some methods for the BST itself.

Example

Implement a BST of ints using the scaffolding below (for simplicity, assume uniqueness of elements)

package tree.bst;
 
/**
 * Simple Binary Search Tree that holds integer values. For
 * extra simplicity, we'll assume that no duplicate values are
 * stored within, for now.
 */
public class BST {
 
    private BSTNode root;
 
    /**
     * Creates a new, empty BST
     */
    public BST () {
        this.root = null;
    }
 
    /**
     * Adds a new int into the BST by first finding its
     * proper location in the existing tree, and then
     * creating a new node containing the int at that spot
     * @param toAdd Integer value to add at its proper spot
     *        in the BST
     */
    public void add (int toAdd) {
        // Case: empty tree
        if (this.root == null) {
            this.root = new BSTNode(toAdd);
            return;
        }
 
        BSTNode current = this.root;
        while (current != null) {
            // Case: insertion is less than current
            if (toAdd < current.data) {
                // Case: OK to insert left
                if (current.left == null) {
                    current.left = new BSTNode(toAdd);
                    return;
                }
                current = current.left;
 
            // Case: insertion greater than current
            } else {
                // Case: OK to insert right
                if (current.right == null) {
                    current.right = new BSTNode(toAdd);
                    return;
                }
                current = current.right;
            }
        }
    }
 
    /**
     * Determines whether or not the given int query exists
     * in the BST by performing binary search on the BST
     * starting at the root.
     * @param query The int with which to perform a membership
     *        test.
     * @return Returns whether or not the given query is stored
     *         within this BST
     */
    public boolean contains (int query) {
        BSTNode current = this.root;
        while (current != null) {
            // Found the query!
            if (current.data == query) {
                return true;
            }
            // Otherwise, keep looking...
            current = (query < current.data) ? current.left : current.right;
        }
        return false;
    }
 
    /**
     * Simple BSTNode containing int data and
     * with two child references: left and right
     */
    private class BSTNode {
 
        int data;
        BSTNode left, right;
 
        BSTNode (int d) {
            this.data = d;
        }
    }
}

We should, of course, test the above:

package tree.bst;
 
public class BSTTests {
 
    public static void main(String[] args) {
        BST tree = new BST();
        tree.add(10);
        tree.add(2);
        tree.add(12);
        tree.add(4);
        tree.add(3);
        tree.add(11);
        System.out.println(tree.contains(10)); // true
        System.out.println(tree.contains(3));  // true
        System.out.println(tree.contains(14)); // false
        System.out.println(tree.contains(1));  // false
    }
 
}

Example

Add a method public ArrayList<Integer> getSortedList(); to the BST class above that returns an ArrayList of Integers composed of the BST’s ints in sorted order.

And there you have it! A nice implementation of a BST.

---

Balancing Binary Search Trees

Question

Say I’m inserting ints into a binary search tree. What property of this input list of ints will cause insertion to take O(n) time?

Answer

When the insertions are done one at a time and arrive in sorted order.

Question

Suppose I do create a binary search tree using the above detrimental insertion ordering. What efficiency guarantee on what operations does my BST lose?

Answer

I lose log(n) performance of search and insertion!

Well, as it turned out, that worst case insertion was problematic… and the more linear, and less tree-like binary search trees became, the less efficient their search became too!

So the eggheads of yore considered making an algorithm that would keep the tree balanced such that any insertion won’t make the tree become too linear.

Definition

Balancing binary search trees is an operation that prevents them from becoming too linear, thus securing the performance guarantees we expect in a BST.

There are a variety of different ways to keep trees balanced and maintain the log(n) search guarantee for binary search trees; we’ll examine a couple now.

AVL Trees

Definition

AVL trees have 1 property in addition to those of binary search trees: the heights (depths) of any two of a given node’s subtrees must differ by at most 1.

This enforced property makes sure that the depth of any subtree never becomes too linear, which would degrade the efficiency of search and further insertions.

Toolkit

This balancing act takes O(log(n)) time, which means that, upon insertion of any new value, we get (O(log(n) + log(n)) = O(log(n))) time; a greater overhead, but same complexity class.

1. Firstly, we keep track of a “balance factor” at each node which is equal to: balance = height(left_subtree) - height(right_subtree)
2. A tree is out of balance if its balance factor is greater than or equal to 2, or less than or equal to -2.
3. If a tree is out of balance, then we use the following balancing algorithm:

// Code skeleton for C++, edited from Wikipedia :)
if (balance_factor(L) == 2) { // The left subtree
    Node* P = left_child(L);
    if (balance_factor(P) == -1) { // The "Left Right Case"
        rotate_left(P); // Reduce to "Left Left Case"
    }
    // The Left Left Case
    rotate_right(L);
 
} else { // balance_factor(L) == -2, the right subtree
    Node* P=right_child(L);
    if (balance_factor(P) == 1) { //The "Right Left Case"
       rotate_right(P); // Reduce to "Right Right Case"
    }
    // The Right Right Case
    rotate_left(L);
}

So what does it mean to “rotate” a tree, you might ask?

Definition

A rotation promotes a child node to the parent, and denotes the parent to a child node depending on the direction of the rotation. Subtree structures are maintained.

[

(credit to Wiki for image)](http://en.wikipedia.org/wiki/AVL_tree)

Above, a right / clockwise rotation would start at the right image and finish with the left image.

Similarly, a left / counter-clockwise rotation would start at the left image and finish with the right image.

Notice that the subtree structures are maintained with each rotation.

Now that we have the basics of rotation down, let’s look at an example inserting into an AVL tree:

Neat!

So remember: AVL trees balance subtree heights, not necessarily just the number of nodes per subtree!

Definition

Let’s examine a visualization here (click me)

Red-Black Trees

So, the one problem with AVL trees is that with a lot of insertions, you’ll have to keep performing the balancing overhead any time a branch becomes 2 depths or more greater than a parent.

Red-Black Trees said, “Hey, let’s not do all of this rebalancing nonsense all the time; everything’s chill until the path from the root to the farthest leaf is no more than twice the distance from the root to the closest leaf.”

So, without diving deep into how it rebalances when this distance is exceeded, let’s just look at some properties of red-black trees lifted from wikipedia:

• A node is “painted” either red or black (a boolean flag)
• The root is black
• Every red node must have two black child nodes
• Every path from a given node to any of its descendant leaves contains the same number of black nodes.

What we end up with is a binary search tree that still ensures O(log(n)) search without the need to so frequently rebalance.

Definition

Let’s examine a visualization here (click me)

Summary

Remark

You do not need to know all of the specifics of these tree balancing algorithms for this course, though the following are important ideas:

• Tree balancing is important to prevent binary search trees from becoming too linear and thus losing their (O(log(n))) performance for insertion / removal.
• Tree balancing algorithms exist in a number of different formats, but all of which deliver a balanced tree upon addition in cost that’s at most (O(log(n))).
• One such method is through the AVL tree’s specification of a balance factor at every node that is updated upon a recursive addition of Nodes to the tree, and serves as a trigger for rebalancing whenever subtree depths differ by more than 1.

---

Trees and Recursion

Something we might notice about the above balancing algorithms is that they’re implemented recursively, and that the addition mechanism is implemented recursively as well.

Before we see how to do so, let’s take a look at a small recursive tree method on our basic BinaryTreeNode class (i.e., the non-search tree version from last lecture).

Remark

Guideline 1: Remember that if the signature of the method you’re implementing is not what you want, you can always create a private helper with whatever parameters you please!

Example

Implement the getLeaves method in BinaryTreeNode, which returns an ArrayList of the strings held in the leaves of the tree.

public ArrayList<String> getLeaves () {
    ArrayList<String> result = new ArrayList<>();
    getLeaves(this, result);
    return result;
}
 
/**
 * Recursively performs a preorder traversal on the given Binary Tree to create a
 * List of all leaf values stored within.
 */
private void getLeaves(BinaryTreeNode n, ArrayList<String> result) {
    // Base case:
    if (n == null) { return; }
    // Visit behavior: check to see if we're at a leaf, and add it if so
    if (n.left == null && n.right == null) {
        result.add(n.data);
 
    // Recursive case: not at a leaf, so keep looking deeper
    } else {
        getLeaves(n.left, result);
        getLeaves(n.right, result);
    }
}

Notes on the above:

• Note how the original, public method of getLeaves wasn’t in a format that we wanted, so instead, we created an intuitive helper method in the format that was convenient!
• Remember that ArrayLists are objects, and since objects are passed by reference, we can manipulate the same ArrayList declared in the public getLeaves method as in the private helper.

Remark

Guideline 2: Remember that recursive methods are still method calls that can return some value to the recursive call that called them! We can exploit this to pass back values to previous frames on the call stack.

Example

Suppose instead we wanted to implement the BST’s addition recursively, inspired by the above, we could do the following:

public void addRec (int toAdd) {
    // Set the root equal to the result of the helper
    this.root = this.addRec(this.root, toAdd);
}
 
private BSTNode addRec (BSTNode n, int toAdd) {
    // ...which will either be a new node in the case of an empty tree,
    // or where we've found the insertion spot to be
    if (n == null) {
        return new BSTNode(toAdd);
    }
    // ...and can set the left and right references appropriately depending
    // on where the node should go
    if (toAdd < n.data) {
        // [!] Update balance factor here since it'd go in left subtree
        n.left = addRec(n.left, toAdd);
    } else {
        // [!] Update balance factor here since it'd go in right subtree
        n.right = addRec(n.right, toAdd);
    }
    // Finally, returning n as the current occupant of the parent's reference
    return n;
}

Remark

The recursive tree-balancers discussed above would have their heights and balancing updated in a recursive implementation like the above!

---

BST Applications

Question

Note that one of our stipulations above surrounding our small integer BST was that the values we were inserting were unique. What kind of a data type that you might remember from your Python days / Math classes has no duplicate values?

Answer

A set!

Definition

Sets are a popular application Abstract Data Type that are implemented using Binary Search Trees, where Sets are simply unordered-collections of items where duplicates are not allowed.

Toolkit

The Java Collections Framework Set implemented as a self-balancing Binary Search Tree is known as the TreeSet data structure.

The two most common operations on a Set are really just additions and set membership tests, just like we’ve implemented in our basic BST class above.

public static void main(String [] args) {
    TreeSet<Integer> leafy = new TreeSet<>();
    leafy.add(1);
    leafy.add(2);
    leafy.add(3);
    leafy.add(3);
    // [?] What gets printed below?
    System.out.println(leafy.size());
    System.out.println(leafy.contains(3));
    System.out.println(leafy.contains(4));
}

Debug

Warning: Using ArrayLists for membership tests is pretty inefficient and is one of the biggest novice moves that tanks efficiency!

Example

Consider the following timed unit tests in which we add a bunch of ints to both an ArrayList and a TreeSet and then perform membership tests on each.

public class TreeSetTests {
 
    private static int TEST_SIZE = 200000;
 
    @Test
    public void testArrayListMembership () {
        ArrayList<Integer> arr = new ArrayList<>();
        for (int i = 0; i < TEST_SIZE; i++) {
            // [!] Appending i to ArrayList
            arr.add(i);
        }
        for (int i = 0; i < TEST_SIZE; i++) {
            // [!] Membership tests
            arr.contains(i);
        }
    }
 
    @Test
    public void testTreeSetMembership() {
        TreeSet<Integer> ts = new TreeSet<>();
        for (int i = 0; i < TEST_SIZE; i++) {
            // [!] Adding i to BinarySearchTree
            ts.add(i);
        }
        for (int i = 0; i < TEST_SIZE; i++) {
            // [!] Membership tests
            ts.contains(i);
        }
    }
 
}

Whew, that’s night and day! Once again teaching us that certain tasks (like membership tests) are better suited for certain data structures than others!

Let’s connect the above to some of your last lectures.

Question

What would be the amortized average-case asymptotic performance of both testArrayListMembership and testTreeSetMembership above?

Answer

Analyzing each separately:

• testArrayListMembership:
  • Adding Loop (first loop): arr.add(i); is an (O(1)) operation repeated (n) times in the loop, for a total cost of (O(n))
  • Membership Test Loop (second loop): arr.contains(i) is a linear search through the List, so an (O(n)) operation repeated (n) times, meaning it’s (O(n^2)).
  • Total: (O(n) + O(n^2) = O(n^2))
• testTreeSetMembership:
  • Adding Loop (first loop): ts.add(i); is a (O(log(n))) operation repeated (n) times in the loop, for a total cost of (O(n*log(n)))
  • Membership Test Loop (second loop): ts.contains(i) is binary search on a balanced BST (O(log(n))) repeated (n) times for another (O(n*log(n))).
  • Total: (O(nlog(n)) + O(nlog(n)) = O(n*log(n)))

Notice how, for even small n, the difference between those computational classes is massive in wallclock time!

This isn’t just about theory, it’s about the practical efficiency that can be really costly when we make the wrong choice… but you’ll know better having taken this class!

---