Managing our own performance can be powerful and will be better than someone else doing it.



## What Success Looks Like

When you're successful, you'll know it, and so will your leader?  Or will you?  Or will they?  

When you state what success looks like, it makes it obvious.  It reveals what you and your leader think.  If you can agree upon something put in concrete terms, you have something to work toward.  Without that, you can only hope to accomplish an unknown standard by luck or approximation.

## Aiming at a Goal

We all like feeling that we're improving.  We're likely active in learning and bringing our new skills online.  But there's so much out there -- what do we choose, and how do we focus?

It's easy to be ever learning and never coming to a knowledge of what we really want or need.

Instead of a shotgun approach, we can zero in on something and aim directly at what is worthwhile to us.  Once we've aimed, the effort of our forward motion is focused and efficient.  We gain improvement and are not merely busy.

## Measurement Magnifies

When we honestly reflect on our progress, we learn how we're tracking toward our goals.  What we learn may be uncomfortable, especially if we see more clearly how we're still deficient.  Still, that's empowering to know.  What we observe can also be glorious, when we realize how far we've come and what we've accomplished.

Measurement will spur us onward and help us desire to keep acting in our own best interest.  As Thomas S. Monson has [said](http://tdan.com/performance-measurement/5265), "When performance is measured, performance improves. When performance is measured and reported back, the rate of improvement accelerates.""

## The Personal Touch

Management is so much better tasting when we do it ourselves.  This is compared to some sterile management by objective or some heavy-handed personal improvement program.

If we engage in our own management proactively, it can be on our own terms.  We won't be waiting to be acted upon.  We will establish, work toward, and accomplish the improvements we and trusted friends see we would benefit from.  We will create a personal regiment that is effective for us.

Do you manage your own performance?  Why or why not?

Managing our own performance can be powerful and will be better than someone else doing it.

Personal Performance Management

My first xquery experience has been on the MarkLogic platform.  The project that we just released was written entirely in xquery and on that platform.  As our site continues to gains popularity, we continue to realize how little about xquery we knew or know.  Sometimes and in some places, the site is just really not that performant.  "But I thought MarkLogic/xquery is super-scalable," some exclaim indignantly.  If you do it right, an Oracle relational database can be made to scale.  Done wrong, a MarkLogic database can be made to not scale at all.  There's a lot to be said about knowledge of the platform, the language, and how to wrestle it to do your bidding.  Here are a few optimization ditties that I've collected as of late that might help in your future xquery dev.



* Flatten your query - This means push your where clause into the xpath at the beginning of your FLOWR statement, limiting the amount of work you have to do once you're inside the for statement.

* Limit your data set - Do whatever it takes.  You just want to get to the smallest set of data possible in the most direct way.  For us, that has meant putting the most specific predicate first or keeping active datasets in a collection (king of AoP-like, not affecting persisted XML data)

* Query once, reference later - This comes pretty naturally in a, say, Oracle world, because it takes so much work to get down to the persistence layer and retrieve the data.  In xquery, the data sits so much closer to your app layer, giving you plenty of easy access.  Just think of it as taking another needless roundtrip to the database every time you want to run a little xpath action.

* Favor xdmp:estimate() - If you can, use xdmp:estimate() because it's faster.  And, it's accurate if counting at fragment boundaries (doc roots).

* Beware calculations - "select count (id)" might work snappy quick in sql, but do the same thing in xquery, and depending on your xpath or where clause, it's gonna work as slow as snot.  The biggest difference?  Probably that documents may need read into memory in their entirety for even a simple result like a count.  Lots of times, it might be good to design for pre-calculations, that happen before the calculation is actually even needed.  For instance, you could put all id's for docs in a certain state inside of a single doc, which is much cheaper to query than many.

* Know your indexes - I'm definitely still learning this one:  which ones are being used implicitly w/o being setup by an admin.  Here's a pearl that we just picked up: Range indexes will significantly increase the performance of your order by clauses that use those indexed elements/attributes (including fn:distinct-values()).

* Limit logging - By nature, file i/o can be expensive.  If you want to keep it in, but turn it off, protect it with a conditional (it's a lazy calculation and hopefully a cheap one too).

* Beware the double slash - This is a given, even for the xquery newb.  This is tree traversal galore.  Then again, don't be afraid to use it, when helpful, for data sets that are always guaranteed to be known and small.

* Utilize search functions - MarkLogic, for instance, has many performance optimizations built into its search capability.  Sometimes it will be more performant to use a search function as opposed to raw xpath.  For example, if an element personId is supplant deep in the bowels of an xml document, requiring xpath muscle to get in, find docs with a particular personId, and return the set, you could do something simple like cts:search(/doc,cts:element-word-query(xs:QName("personId"), $personId)) that could be legions faster.

Those are some dev tips, here are your tools, really quickly:

* Known and love [CQ](http://developer.marklogic.com/code/cq) and its profiler

* xdmp:query-trace() and  xdmp:query-meters()

* Some code to generate large amounts of data for you to test on

Finally:

* There are a few more tips in the Priscilla Walmsley [XQuery](http://www.amazon.com/XQuery-Priscilla-Walmsley/dp/0596006349/ref=sr_1_1?ie=UTF8&s=books&qid=1278002233&sr=8-1) book, chapter 15.

- Performance guide available at [developer.marklogic.org](http://developer.marklogic.com/docs).

Happy coding.  And remember, it's always possible to write bad, unperformant code.

My first xquery experience has been on the MarkLogic platform.  The project that we just released was written entirely in xquery and on that platform.  As o

Xquery Optimization Tips

I was running an SQL query today and it was sooooo slow.  So slow, in fact, that it never returned.  I asked the DBA, Reed, who built the table what might be up, and he informed me that it was not indexed.  And proceeded to show me some cool stuff I could do to actually get my query to return.  In the end, it was a comparison between the "not in" operator and a "left join".



My original query was thus, names changed to protect the innocent:

```sql
select count(*)
from   temp_legacy_attachments i
where  i.person_id not in (
  select m.legacy_person_id
  from   new_attachment a
  ,      person p
  where  a.person_id = p.id);
```

I was trying to query the temp_legacy_attachments to get all rows that didn't have a record in the new_attachments table.  It never returned, and so Reed told me to give this one a try:

```sql
select count(*)
from   temp_legacy_attachments i
left join (
  select m.legacy_person_id
  from   new_attachment a
  ,      person p
  where  a.person_id = p.id) ea on ea.legacy_person_id = i.person_id
where ea.legacy_person_id is null ;
```

So, instead of using "not in" a set, I select all the legacy rows, then outer join to the new_attachment rows and filter where a column on the new attachment set is null (it's the smaller/less-available set).

I thought it was pretty sweet.  No magic bullet, though, as Reed tells me that there is a fair amount of debate over the performance difference between the two methods.  You just have to try it and find out.  For me, in this case, the outer join was more awesome.

### Update

Another savvy DBA, Bill, has graced us with another method yet:

```sql
select sum(cnt)
from (
  select p.legacy_person_id
  ,      count(*) cnt
  from   person p
  join   temp_legacy_attachments i on i.person_id = p.legacy_person_id
  where  not exists
    (select null
    from   new_attachments a
    where  a.person_id = p.id
    and    a.created_by = 'LEGACY_MIGRATION')
  group by p.legacy_person_id)
```

I was running an SQL query today and it was sooooo slow.  So slow, in fact, that it never returned.  I asked the DBA, Reed, who built the table what might b

Not in vs. Outer join Performance

Often when coding, we use a single local variable multiple times, overwriting the value many times.  It is considered good practice to move the variable out of the looping overwrite and into the smallest scope of code that is run once.  But, this makes the code a little bit less concise.  So, how useful is it, anyway?  I wanted to run a few little tests and see if there's really a noticable difference in performance.  



### Experiment

My primitive experiment was to create two different Java programs where I kept declaring the variable in one and where I declared it once and then assigned in multiple times in another.

```java
class TestDeclareOnce {
  public static void main(String[] args) {
    int x;
    for (int i = 0; i < Integer.parseInt(args[0]); ++i) {
        x = i;
    }
  }
}

class TestDeclareMany {
  public static void main(String[] args) {
    for (int i = 0; i < Integer.parseInt(args[0]); ++i) {
      int x = i;
    }
  }
}
```

And run like so:

```bash
java TestDeclareOnce [num-times-to-loop]
```

I wrote similar versions for java.util.List's, trying to determine whether or not constructing and assigning a larger object would change the pattern or not.  Of course, the declaration of an object is only a reference to somewhere in the heap, so I'm not sure what I was expecting, but I tried it all the same.  Those classes were very congruent in form to the first two, differing in this form:

```java
List list = new ArrayList(i);
```

One of my initial theories was that the Java compiler would optimize the difference away, making the byte code the same.  I diff'ed the resultant 2 bytecodes, however, and they are indeed different.

So, I instead ran these a few times and tried to determine the difference in time to completion.  I used the Unix 'time' function to record running time.  Who knows how accurate that is, but it was handy at the moment.

### Results

<a href="https://picasaweb.google.com/lh/photo/LvQyLtgitOy4XVWZYjX7LQ?feat=embedwebsite"><img src="https://lh6.ggpht.com/_5XZCKcD6--c/Sp6-v4GovRI/AAAAAAAAIWg/KlHJpd4HQsI/s400/VarDeclarationSpeedResults.png" /></a>

I didn't run this test many times and look for the average or mean, so there are some outliers here.  But, I think there's a fairly obvious, if not conclusive, pattern.  Defining a variable outside the loop makes a bigger difference the more times you loop.  Even then, some of the time the results are very close or even contradict the previous statement.  Data sets used here weren't very large either, so the difference is less drastic.  

The conclusion?  It's not going to make a big difference whether you declare your variable inside or outside the loop when dealing with a small number of loop iterations.  Here, small has been demonstrated to be up to 100 million.  There may be some situations in which it matters more.  What has been your experience?

Often when coding, we use a single local variable multiple times, overwriting the value many times.  It is considered good practice to move the variable out