Locklin on science

The bias ratio: lousy at fraud detection

Posted in finance journalism, fraud by Scott Locklin on April 15, 2009

The bias ratio is a method used to flag hedge funds for fraud. It’s most famous recent use was “unmasking” Bernie Madoff’s fraud, at least the Financial times wrote an article which asserted this. Of course, it was done post-facto, and the article is based on a marketing document by Risk Data, but whatever; it’s worth a closer look at what they say, and how well it holds water.

How is a fund manager most likely to commit fraud? Well, funds often post monthly returns. Chances are good, the fraudulent manager will tend to underreport negative monthly returns. They’re mostly judged by things like the number of negative return months; silly, but that’s how things work. What is the bias ratio, exactly? You could read the Wakipedia explanation. It is an uncharacteristically good explanation, and I used it to code up my version of this in R, but it is also over wordy, and misses the point in some ways. My shorter better version: take the standard deviation of all the monthly returns for the history of the fund (or some subset over a time period which makes sense). Now, look at one standard deviation of returns in the positive and negative directions, centered around zero. Count the number of months of returns which are within one standard deviation of the over all returns in the positive and negative directions and take their ratio. You’d expect something which has positive returns to have a positive bias ratio. You’d expect something where the manager was cheating to have a large positive bias ratio.

There are a couple of problems with this measure. What happens if there is no market for the assets under management? Markets are mechanisms for price discovery. In an illiquid market, you might have a large spread of prices to mark to. In a very illiquid market you might have to do something like mark to model. As such, something like a MBS fund will have a huge bias ratio. There is no MBS market, the way there is a market for AT&T stock. The net asset value of an MBS fund or any other illiquid fund is going to be smoothed as a result. Other funds that specialize in merger arbitrage or distressed assets may also suffer from this. Equities funds are not supposed to have this problem. Equities usually have good market prices, and they follow a random walk. They should give low numbers for bias ratio, just like a stock which skews positive does. I’m a scientist, so I don’t take people’s word for anything. So I went and had a look at what the FT asserted. Lo and behold, if I compare Bernie to some long short equity funds the way riskdata does, he does indeed rather stand out in terms of bias ratio (click on the graph for a close up):


longshort-bernie

One problem which springs to mind is, a fund with simply very good returns (or at least very few negative ones) will also have a large bias ratio. One of the first things I looked at was the EDHEC data on different fund strategies. You can see that, while Bernie’s bias ratio is high compared to most of these strategies … equity market neutral has a value which is higher still. I have no idea what EDHEC used for their equity market neutral fund, but it’s got a good bias ratio basically because they had very few down months. I don’t have data for Renaissance’s Medallion fund, but it will also have a very high bias ratio for the same reason. So in a way, bias ratio as a red flag is a lot like using the Sharpe or Sortino ratios as a red flag. Possibly a useful component of a scoring system, but the strong assertion made by the Financial Times that this is a useful stand-alone quant screen for fraud doesn’t hold water.

edhec-bernie

Now, if I were running any kind of quant screen on old Bernie, I’d try the very obvious idea of comparing the results to similar strategies to the one he was claiming. The claim was that he was using something called a “split strike conversion;” a known options strategy. The other, hidden claim was that he was illegally front running his own customers through his market making business. So, it would be logical to compare his returns to those of a basket of companies which do things like this. Nobody actually trades split strike conversions as their main strategy, but lots of people make money selling options. High frequency guys probably look a lot like front runners, since what they do is often effectively front running. As it turns out, options related strategies of different kinds also seem to fall roughly into the range that Madoff’s fund had for a bias ratio. I’m not sure why it is that options strats are high in bias ratio; perhaps they’re marking to model. Perhaps most options strategies are taking a position on volatility; volatility is pretty well modeled by a smoothing process. If you’re taking a position on something which can be modeled with exponential smoothing (GARCH, whatever; it’s just fancy exponential smoothing), the returns are going to look smoothed too. So, perhaps these option funds are shorting volatility -that probably would look like smoothed returns if you do it right. Or perhaps options traders tend to be really good risk managers because of how they think about what they are trading.

I’ve found an exceptional fund which trades out of the money index options and “fails” this test with more than double Madoff’s score on bias ratio: Zenith Resources. They show no signs of being a potential fraud. They answer emails and are very public, their strategy is known to be very profitable (split strike is … not so profitable), they’re running a reasonable amount of money for what they do (unlike Madoff’s fund, which was absurdly huge), their returns are sensible, they don’t need any super powers to make money, and they seem to be closed to new investors -not something that fraudsters tend to do. In the extreme case of their strat where you only sell really far out of the money options, their strategy can be thought of as a bet against western civilization collapsing. That’s a pretty good bet. I mean, it’s mean reverting, but it won’t matter if you lose the bet, financially speaking. You’ll be able to challenge the counterparty to a duel. It’s also effectively a short on volatility, which is likely part of the reason the returns are so smooth. Sure, a “Black Swan” like October’s insanity can kill a business like this, but that’s where skill and quantitative risk management come in. Even in the “Swan case,” they will still have a high bias ratio, since “Swans” by definition happen more than one standard deviation out. As you can see, there are a couple of other funds which also have a comparably high bias ratio. They all sell out of the money options as well; taking a position on volatility the way Zenith does. Bernie’s bias ratio is not at all exceptional in this cohort of options trading funds. Bernie stank for lots of other reasons, as outlined in Markopolos’ amazing document of 2005. But bias ratio … not so informative.

bernie-vs-options

So, nice try, Financial Times and RiskData guys, but your quant screen for fund fraud doesn’t really work. It might be a useful component in a screen, but it doesn’t cause Bernie’s fraud to leap out in any obvious way, unless you lump him in with a bunch of funds which aren’t at all like what he was claiming he was doing. Put him in with other options guys, even delta neutral funds, and Bernie’s bias ratio doesn’t look very remarkable at all. In a way, bias ratio is a performance measure of the centroid of returns around zero. If you’re going to start doubting everyone with good performance metrics, well, that rather descends into self parody. Bias ratio mostly makes me want to go sell uncovered out of the money puts to schlubs who think I’ll actually pay them if Ragnarok happens. I can practice my dueling skills while waiting for Fenrir to break his chains in the meanwhile. Keep that wolf at bay, gents.

odin_and_fenris2

Advertisements

4 Responses

Subscribe to comments with RSS.

  1. […] looks like a plain old bear in a bull market. There are far more successful and long lived funds, as I have already pointed out which trade the inverse of the Taleb “swan strategy” at very high Sharpe ratios. They […]

  2. Wee Kang Chua said, on July 21, 2009 at 7:46 am

    Hi,

    Come here by way of alea. Really like your writing – intelligent & insightful.

    I assume you’ve looked at Zenith’s numbers. Their index options program returns are nothing short of amazing. Only down months so far were understandably Feb 07, Jul 07, and Oct 08.

    But it seems like they stayed out of the market in Nov and Dec 08. And the returns from 09 so far seem to have fallen short compared previous years.

    But it’s still quite unbelieveable how they managed this basically through being short vol. Any ideas how to replicate their strategy?

    • Scott Locklin said, on July 21, 2009 at 5:13 pm

      I have seriously considered attempting to replicate Zenith as a leg in … well, I might end up talking about it, because I don’t know how to replicate his returns. Strap a Zeneth together with an Amplitude (I do know how to replicate what they do), and you can get something which is greater than the sum of the parts.
      By the way, the proprietor of Zenith answers email. He strikes me as a righteous dude.

  3. […] looks like a plain old bear in a bull market. There are far more successful and long lived funds, as I have already pointed out which trade the inverse of the Taleb “swan strategy” at very high Sharpe ratios. They even win […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

%d bloggers like this: