Locklin on science

The economist; staffed by statistical buffoons?

Posted in finance journalism by Scott Locklin on June 22, 2010

I came across this in one of the Economist’s “Free exchange” blogs: (via NP)

The funny thing is that according to Sebastian Mallaby’s new hedge fund history, “More Money Than God”, the willingness to explore unexplained correlations is what sets Renaissance apart from other quant funds (full disclosure: Mr Mallaby is married to The Economist’s Economics Editor). Where other funds might recruit employees with financial or economic backgrounds and have them test hypotheses against data, Renaissance employeed thinkers who had spent the bulk of their career in non-economic analytical fields, like mathematics, physics, and astronomy. Once at Renaissance, those thinkers would build data-processing models without any preconceptions about what should cause what, when. The firm’s advantage is in its willingness to trade what doesn’t necessarily make sense.

One of the many truly funny things about this is the implication that, say, some dude with a financial or economic background is better at hypothesis testing things than a physicist or applied mathematician. Here’s a thought for “R.A.” -maybe scientists think to look for actual -rather than spurious correlations in places that economists don’t? And maybe they don’t look for spurious correlations in places economists insist they must be? Economics isn’t much of a science; I’d characterize it as somewhere around the four humor theory of medieval medicine. Not completely wrong, but not particularly right either, and a gratuitous over simplification of how things really work. The type of reasoning that goes into it isn’t necessarily wrong, but it’s obviously pre-scientific. Some medieval doctors were probably very good doctors, and some traders are probably very good economists, but it’s a lot easier to pick a good medieval mathematician than it is a good medieval doctor. And you know you could train the math guy to be a doctor. That’s kind of the idea of hiring science nerds to do applied economics.

Does RenTech preferentially select people without financial knowledge? Yes, I know for a fact that they do, because a former Medallion alum laughed uproariously at the fact that I was taking the CFA (I didn’t bother taking the test in the end: thanks, doc: you saved me some money). Does that mean they bet on things like astrology applied to futures markets? I kind of doubt it. In a real science, people like to get the right answer: not the ideologically correct answer. That’s why RenTech hires scientists, and people like Natural language translation experts. There is a right answer, and those people are the types who go out and find it, based on minimal clues.

But having the guts to trade relationships no one else can understand or explain would be one way to consistently beat the market over a period of two decades.

Um, no; that’s how you lose all your money in two weeks.

One of the many horrible things about reading this sort of thing: The Economist is probably the best magazine in the world. Read by leaders, diplomats, CIA operatives, and bigshots the world round. If they’re this laughably wrong in such a simple matter, how can you believe anything they say? I mean, why would anyone think RenTech looks like a Thomas Dolby video?

About these ads

23 Responses

Subscribe to comments with RSS.

  1. William O. B'Livion said, on June 22, 2010 at 2:21 am

    ;
    ; I kind of doubt it. In a real science, people like to get the right answer: not
    ; the ideologically correct answer.
    ;

    Does anyone do real science any more?

    • Scott Locklin said, on June 22, 2010 at 3:18 am

      Chemists! A chemist who doesn’t want to be a real scientist will call himself a “nanotechnologist” or something.

  2. Dirk Eddelbuettel said, on June 22, 2010 at 3:01 am

    But see http://feeds.felixsalmon.com/~r/felix-all/~3/-1frEJd–yc/ for a somewhat conflicting description of the model search process at RenTec.

    • Scott Locklin said, on June 22, 2010 at 3:18 am

      Yeah, Salmon’s review was linked/contrasted in the economist blog, and sounds a lot closer to what I know about those guys.

  3. maggette said, on June 22, 2010 at 12:01 pm

    During my studies I took a econometrics seminar. The topic was market micro structure.

    My job was to implement and estimate a model published by the prof and test it on another market. Since they think their students are stupid, they provided me with the data and some GAUSS code. The parameter estimation was done via ML, and the according MLE GAUSS function was presented to me as some “arcane” black box, that should be handled with care (don’t play around with the options).

    I transfered that stuff to MATLAB and had a closer look on what I was actually doing. The model had five different parameters and I had way way way too few data points to estimate anything.

    I simulated the theoretical model (hence I knew the real parameters) to get a feeling of how much data I actually needed to estimate the parameters . Defensively estimated I need at least a 10^3 times larger data set than the one the prof used in his publication.

    I made the mistake pointing that out during my presentation (I even created a matlab avi that showed the ML Surface moving as a function of the sample size). He had to give me an A….but he hated me ever since.

    I also presented some papers from Bouchaud and Farmer to them (both Physics guys) on the same topic. My profs assitants dissmissed that stuff (of course they weren’t aware of the work until I gave them the papers) because it “lacked the economic intuition”.

    I replied that I don’t know if that (lacking econ intuition)actually is a bad thing.

    The level of ignorance and “group think” amongst economists I met is incredible. How can you be so sure about your theory when your models lack any predictive power?

    I am also confused by some of the rigor that is so important to them. They write down a list of absolutely hilarious assumptions that is several pages long and use a data set of 50 points GDP data or something…..and than laugh about a guy who used an estimator that (given the assumptions) isn’t BLUE anymore ?!?
    I don’t get it.

  4. maggette said, on June 22, 2010 at 12:04 pm

    The level of ignorance and “group think” amongst economists I met is incredible. How can you be so sure about your theory when your models lack any predictive power?

    I have to put a “most” into ” amongst economists I met is incredible. ”

    I have also met some very smart (or at least much smarter than me) econ guys who weren’t that religious.

  5. DoobDoob said, on June 22, 2010 at 6:04 pm

    @maggette: I am intrigued and confused. Your description of using MLE to estimate models of microstructure noise in financial markets sound like the kind of procedures devised by Yacine Ait-Sahalia over at Princeton. These are not simple models and I would be very keen to see how you came about your calculations of 10^3 times the data-points required, especially as these models are applicable to high-frequency returns where the sampling rates and the number of points are typically very high (in-fill asymptotics) and I would assume you would be supplied with such data. Could the fault lie in the optimisation algorithm you employed?

    Secondly, I would love to see what these hilarious assumptions are and where you found them. Most of the work of econometricians is to get things to work under minimal assumptions on the data-generating process. Examples of unnecessary assumptions would be lovely.

    Further, under no reasonable assumptions on time-series data (as GDP is) could you end up with a BLU estimator – so again, I am intrigued by your tirade.

  6. maggette said, on June 22, 2010 at 11:35 pm

    I don’t know all his research but I regard Ait Sahalia as quite similar to the Andrew Lo type of econometrician. That’s good stuff. And some of his stuff I read I can at least imagine that sombody might use it.

    It is a while ago but the model I did was about estimating the “Probability of informed Trading” (PIN). I think the basic paper was by Easley and O’Hara.
    I don’t find them on the net. But I think I have them somewhere. If you want to give me an email I would send them to you. Than we could discuss absolutely senseless assumptions if you like…

    This one is quite similar:
    http://www.eurojournals.com/irjfe_33_02.pdf

    We had to do it simultaniously in two markets and compare the PINs. And that model was and is utter nonsense. I lack confidence on a lot of things, but I got a very strong opinion on that one. This stuff is bad science and absolutely worthless in practice (as Scott pointed out…that does not imply that nobody uses it). I had a lot of epmirical evidence that some assumptions are straight up violated (For example the model needs independece of buyer initiated trades and seller initiated trades…across time as well as accross sells/buys. This was not the case (significant autocorrelation on many lags and cross correlation….”well….so it is”. ).

    “10^3″
    I simulated the according model (so the parameters I want to estimate are my input). Draw 10000 samples of the same size. Run my ML estimation for each sample and compare how close I came and how large the error is (on average across the 10000samples). Then draw another 10000 samples with an increased size….. I don’t claim to have scientific result here, but for artificial samples of size close to my original sample, the estimates were nowhere close. I increased the sample size to 10^4 of the original size. From 10^3 on I came at least close to my parameters (if I recall it right).

    I made that BLUE/GDP stuff up (GDP was the first what came to my mind when I thought about “few data points”). I think you know what I mean. There is a lot of rigor about the asymptotic properties or finite sample properties of OLS/ML/GMM estimators, which is fun math, but I don’t see what it is good for. I just don’t see it. And like I said: I pointed out the problem with non independent data….this was just shrugged off….

    “Could the fault lie in the optimisation algorithm you employed?”

    I have to admitt that might be the case. For example it reacted quite strange to changes in the step width. If I recall it right I tried at least two different algos. I will have to look that stuff up.

    If you are really that interested I will have to dig a little bit in my “archiv”.

  7. [...] Scott Locklin, for one, thinks that’s ridiculous. Trading a relationship no one can understand or explain, he says, is “how you lose all your money in two weeks”. True scientists, says Locklin, of the type hired by RenTech, are trained to look for actual rather than spurious correlations, and to be able to tell the difference. [...]

  8. deniz said, on June 27, 2010 at 5:10 pm

    It’s a different approach. Starting point for economists is a concrete model or, lacking that, an intuitive argument/story. Whereas ‘quants’ start with the data and then build a story/model around the results. I remember DE Shaw describing their approach as the latter. IF we are talking about robustness, the latter approach would be more prone to a number of biases especially if the intuition/story is thin.

    As far as a mathematician being better at medicine, let me quote Feynman:

    “I believe that a scientist looking at nonscientific problems is just as dumb as the next guy.”

    Economics is a social-science. On the one hand you can’t say that it is not a science and at the same time say that scientists would be better at it. If human emotions govern behavior, maybe psychologists would be better at it…

    • Scott Locklin said, on June 29, 2010 at 9:43 pm

      Well, I’m a huge fan of that Feynman quote, but my point stands; medieval mathematicians had measurable skills which carry over to other practices, like medicine. Gerolamo Cardano, for example: not only a great mathematician, but also a pretty good doctor. Could you say the same about a contemporary “good” theologian?
      I respect a lot of what economists do, but I rarely respect how it gets done, and I have zero respect for the gaping credulity these guys are granted in modern society. Physicists are taking cracks at these problems. By all accounts, guys like Sornette and Bouchaud have been pretty successful; I’ll be talking more about him at some point in the future.

      • deniz said, on July 3, 2010 at 5:36 pm

        I don’t know many medievael theologian doctors, but my limited knowledge of history suggests that some science was conducted in monasteries and quite a few intellectuals were theologians/priests as their first occupation. But generalizing from a few names one can google up seems hardly scientific :) I don’t see the mechanism that would make mathematicians good doctors. Is it because they are objective, smart and analytical? Those would seem very general characteristics, that you could easily say the opposite: doctors would make good mathematicians because doctors are objective, smart and analytical. I am not a mathematician but I do know that it requires a distinct talent just like in music or sports. But I don’t see how being good at for instance proving abstract theorems in number theory would make someone good at diognsoing a certain disease with limited knowledge in highly uncertain situations. Perhaps there is a mechanism but it is not obvious.

        I do not know Bochoud, but I do know some of Sornette’s work. If analyses is based purely on data, it’s hard to see why some mechanism that seems to explain some phenoneman in the physical world (like geophisycs– another specialty of Sornette), should explain stock prices. Ideas from physics provide nice anologies or metaphors for financial phenomenon but you need some intuition, a story, or in other words an economic model to make sense:) And it seems the reason physics analogies are used in finance is because most quants are ex-physicists. I must have seen at least three different books with the title ‘quantum finance’ as if somehow that phrase makes any sense. Perhaps if zoologists were better programmers and had somehow ended up on wall st, we’d be talking about how prices were like migration patterns of birds. The same criticism you mentioned should apply to the so called econo-phycisists.

        But, as I quoted friedman, it’s best to judge models not by their assumptions but by their results. But in that regard Sornettes record is abysmal – though one should credit him for at least publishing his forecasts.

  9. deniz said, on June 27, 2010 at 5:17 pm

    PIN is an interesting measure. The model is very simple built on simple assumptions. A number of papers built on the original model by introducing non-linearity, realxing some of the distribution and independence assumptions. None of them really improved on the results of the original model. As friedman said judge a model not by its assumptions but by its results. And there is something to be said of simplicity…

    The model is of course incomplete but the measure seems to be priced in the cross-section of equity returns and seems to hold up on key event dates. Is it the whole story? Of course not. Does it fail? Absolutely. Does it tell us something useful about information content of trades? Yes

  10. deniz said, on June 27, 2010 at 5:18 pm

    Btw, looked at some posts here, very nice blog.

  11. maggette said, on June 28, 2010 at 8:37 am

    “Does it tell us something useful about information content of trades? Yes”

    Then I missed the critical point? What does this model tell me about the information contend of trades?

    Thanks

    • deniz said, on June 28, 2010 at 6:05 pm

      What does this model tell me about the information contend of trades?”

      The key question is if the measure captures liquidity or information asymmetry. Empirical evidence suggests the latter. What people find surprising is how well the measure works given its simplicity. There are also interesting asset pricing implications which have been studied empirically. Easly and Ohara along with Hvidjaer have shown that PIN is priced in the cross-section of equity returns. Haito Li does the same for treasuries. There is new work on it using TRACE data for corporate bonds…

      An interesting practical application would be to see if the measure helps improve performance of pairs trading strategies. There is some evidence that suggests that convergence is faster when the initial divergence is unrelated to fundamental news (earnings announcements, press releases, etc.) but due purely to liquidity. So, if pin is able to tell us something useful about information, then it should help improve returns on pairs trading. I can provide the data if there is anyone here willing to look at it…

  12. maggette said, on June 28, 2010 at 8:51 am

    Sorry, me agian

    I couldn’t say it better than this. Please read from
    “3.1 Information and fundamental values”
    http://www.santafe.edu/media/workingpapers/08-09-043.pdf

    regards

    • deniz said, on June 28, 2010 at 6:05 pm

      The paper is 100pgs long :)

  13. deniz said, on June 28, 2010 at 6:04 pm

    What does this model tell me about the information contend of trades?”

    The key question is if the measure captures liquidity or information asymmetry. Empirical evidence suggests the latter. What people find surprising is how well the measure works given its simplicity. There are also interesting asset pricing implications which have been studied empirically. Easly and Ohara along with Hvidjaer have shown that PIN is priced in the cross-section of equity returns. Haito Li does the same for treasuries. There is new work on it using TRACE data for corporate bonds…

    An interesting practical application would be to see if the measure helps improve performance of pairs trading strategies. There is some evidence that suggests that convergence is faster when the initial divergence is unrelated to fundamental news (earnings announcements, press releases, etc.) but due purely to liquidity. So, if pin is able to tell us something useful about information, then it should help improve returns on pairs trading. I can provide the data if there is anyone here willing to look at it…

    • maggette said, on June 30, 2010 at 1:50 pm

      Hej Deniz,

      thanks for your reply. You don’t have to read the whole paper. Just the 3.1 part. That pretty much sums it up.
      “What people find surprising is how well the measure works given its simplicity.”
      Could you please give me some insight by what measure people judge that it “works”? I mean you need some way to measure the predictive power of the model.

      I have very very fundamental problems with that approach in general. The ML estimation is the least severe of them.

      I don’t think that you can measure “information asymmetry” using the PIN. Not at all (if we still talk about the Easley/O’Hara stuff), because I disagree with the definition of “information” here.

      The model relies on the concept of a fundamental value (singular!!!) of an asset and that these values are driven by exogenous factors and that the according asset prices converge to this value (in this model at the end o each day!!!). That is something close to never questioned in the realm of the financial econometrics discipline called asset pricing. I don’t believe in it and you’ll get a hard time proving that it exists.

      In this model information is knowledge about this fundamental value and hence the future asssset price (at the end of each day).

      I also doubt the “intention” implied by buying above or below midprice. You won’t distinguish a liquidity taker from an informed speculator this way. Just don’t see any reason (besides “economic intuition”) for that. All I know (not that much) about execution algorithms make me doubt it.

      I didn’t test it, but I almost guarantee that the PIN won’t give you any sensible information on the speed of convergence of a spread. I never thought about it, but I think there is are a lot of much more direct and straightforward measures on the impact of news and announcements on the speed of convergence than estimating a PIN using some crazy model.

      Just my opinion
      cheers

      • deniz said, on July 3, 2010 at 5:14 pm

        “Could you please give me some insight by what measure people judge that it “works”? I mean you need some way to measure the predictive power of the model.”

        Well, if the model is non-sense, then it should just produce noise and there would be no reason for it to be priced in equity and bond markets. The papers I mentioned earlier show that it is. The question has been whether the measure captures liquidity or information asymmetry. To answer that question a number of people have looked at information through company characteristics, different exchanges, analyst coverage, corporate events, etc. Easley and Ohara built part of their careers on this, so you can look at a few of their papers. Someone I know did the opposite: instead of information events he looked at liquidity events with no information (terrorist attacks), and also found evidence consistent with the theory.

        “The model relies on the concept of a fundamental value (singular!!!) of an asset and that these values are driven by exogenous factors and that the according asset prices converge to this value (in this model at the end o each day!!!). That is something close to never questioned in the realm of the financial econometrics discipline called asset pricing. I don’t believe in it and you’ll get a hard time proving that it exists. ”

        Asset pricing is a big diverse discipline. Saying something is rarely questioned is not true. You are also reading too much into the notion of ‘fundamental value’. It’s a modeling tool. It’s just meant to capture the fact that there are different states of the world which impact stock prices through their impact on the economics of the firm. For instance in one state BP’s top kill operation works and they plug the leak (which has implications for future income of the firm). In another state (as it happened) the operation fails. You can introduce noise into the relationship stock price = f(income + ….), and you can have different people have different opinions on the probability of success or different levels of information, or different opinions on the impact of the leak on future income, or even different opinions on the relationship between income and the stock price. All but the latter Esaley and Ohara do in their paper with Hvidjaer. Of course we don’t have BP type events everyday, and most trading is driven by liquidity, rebalancing, hedging, etc. and not necessarily by information, but this gives us a framework to think about the issue that some people’s trades are just noise and some contain information.

        “I didn’t test it, but I almost guarantee that the PIN won’t give you any sensible information on the speed of convergence of a spread. I never thought about it, but I think there is are a lot of much more direct and straightforward measures on the impact of news and announcements on the speed of convergence than estimating a PIN using some crazy model.”

        Yes news flow you can measure (like here: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1330689. Information flow, not so much :)

        Be open minded. If you had not known about the cross-sectional effects of PIN., you would have said this crazy would model would not explain any cross-sectional variation :)

        • maggette said, on July 4, 2010 at 7:01 pm

          “Be open minded.”

          I tried to. But implementing the Easly/O’Hara my whole intuition yelled:no, don’t do it:). Maybe I judged a little bit too fast…

          I will have a closer look at the papes you mentioned and come back if I have a better picture on what’s really going on with that approach.
          Thank you so far.

          And thanks for the paper on news and liquidity.
          Mag

  14. [...] Scott Locklin, for one, thinks that’s ridiculous. Trading a relationship no one can understand or explain, he says, is “how you lose all your money in two weeks”. True scientists, says Locklin, of the type hired by RenTech, are trained to look for actual rather than spurious correlations, and to be able to tell the difference. [...]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 288 other followers

%d bloggers like this: