Discussions

News: Algorithmic trading with Twitter

  1. Algorithmic trading with Twitter (41 messages)

    A year ago someone asked me if I'd tried Twitter, "why would I want to know what people are watching on TV?" was my response. Just over a month ago I was asked if Twitter was a viable media for updating our clients about product updates, I'm ashamed to admit I said "no". This week I find that traders are using Twitter as a feed for algorithmic trading, are they mad? My company sell FIX engines specifically tuned to connect to exchanges, places like the CME (Chicago Mercantile Exchange), the ICE (Intercontinental Exchange) and NYSE (New York Stock Exchange), we have 100 odd customers ranging from a couple of guys trading their own money in Monaco to extremely large petrochemical companies in Houston, Texas so it's fare to say we know a thing or two about who's trading what and how. When I saw a press release this week, as part of a finance show (SIFMA in New York), about StreamBase developing a Twitter interface to their CEP engine I initially thought they were either mad or simply going for consumer press coverage. After talking to a few of our customers it seems as though this is actually critical to today's trading. For the last 20 plus years we'd think nothing of using news feeds from Reuters as a source of information, if an oil-well was hit by a hurricane, oil prices went up, if a company founder or CEO died the share-price of the company changed drastically, if a riot broke out in a country it could effect the exchange rate of its currency. Basically, once the news is on Reuters it's official and the prices start to change as soon as the news comes out. If you're going to make money though you need to be the first with the news, if the price is going to fall you want to sell (or borrow if you're shorting) before it falls, not during or afterwards. If it's about to go up you want to buy before it goes up and not after it's too late. So, how do you know if Twitter coverage is real or just Twitter chatter? Well this is where a CEP (Complex Event Processing) engine comes in. You listen to everything and write clever algorithms (functions or methods) that aggregate the news and decide on it's significance. Once you decide it's probably true then you trade on that information. Bear in mind that other people are also watching the same "news" so they are likely to trade too and even if the news is wrong you still need to be trading before everyone else does. So it seems Twitter has become a vital part of my FX (Foreign Exchange) and energies trading world, Streambase were first and for that they deserve some qdos, we'll now see the larger, less agile companies following suite, the nicest form of flattery. Neither I nor my company have any connection with StreamBase, I just thought this might be an interesting subject to discuss. -John Davies- Technical Director Onix Twitter: @jtdavies

    Threaded Messages (41)

  2. Very interesting... I've attended 2 Devnest conferences in the City of London (http://twitter.com/devnest) and quite frankly, I was the only developer from the financial industry... So, i was wondering if I was mad to think about this... One thing that I have learned however is that only a very limited number of "trusted" companies have access to the FULL feed from Twitter (hose-pipe). And I believe these to be the like of Google, Bing (very very limited). Everyone else has access to a limited "selected" feed, it is possible to request a better feed but the whole process is opaque and based on who you know more than anything else (these are my views btw). So, did StreamBase manage to get access to the firehose? I doubt... so the StreamBase algo will make decisions based on a limited set of information... oh, hang on, that is exactly like trading... Very interesting, I wonder what kind of Tweet they push out... (spamming to create a buzz?) Our company is using Twitter but simply to post some server status info... still very low level, I know. Interesting stuff, Benoit Http://www.Appendium.com
  3. Benoit, I'm not sure you'd actually need access to the "firehose" as you put it, you just need a feed to the usual news sources and perhaps a few others and you let the clever logic sort out whether something is worth acting on or not. Of course access to the full "firehose" would be interesting but it's probably a lot more noise than you really need. -John-
  4. Benoit, I'm not sure you'd actually need access to the "firehose" as you put it, you just need a feed to the usual news sources and perhaps a few others and you let the clever logic sort out whether something is worth acting on or not. Of course access to the full "firehose" would be interesting but it's probably a lot more noise than you really need. -John-
  5. During some devnest discussion a "Twitter Manifesto" was discussed in order to clarify some of the processes around Twitter. See http://www.nuxnix.com/images/stories/devnest/mz-twitter-devnest-manifesto-230609-final.pdf For instance, what is the difference between a "selected/reduced" feed and the full feed? I don't know, none of the devnest attendees knew for sure what it meant. You are also limited to the number of people you can follow (unless you are a celeb/cnn/etc) Twitter, rightfully, sees the raw data as their crown jewel but the whole process and ecosystem is still very opaque. Do you get enough information from the standard feed? StreamBase seems to think so, but it is more like a feeling rather than a metrics or set of statistics to prove it. In any case, having some information is better than no information, I agree with your examples and also one could use the iphone 3S example eg factoring in the number of iPhone-related tweets around the launch to realise that they were selling like hot cakes... 1 million units over the weekend as it transpired later on... Useful trading information I'd say. Benoit
  6. As jeff goldblum shows us, even the dead and twitter http://news.yahoo.com/s/eonline/20090630/en_top_eo/131952
  7. I think reuters and other news agencies were already working on these types of services. Not sure if anyone is using it or how. The risk with using tweets is the data is not reliable, so you need some pretty sophisticated fuzzy logic functions to make the news useful. To the best of my knowledge, none of the commercial products have built-in fuzzy logic or real inference capabilities. Whether it's a gimmick or practical in "real life" is yet to be seen. my bias 2 cents peter
  8. The risk with using tweets is the data is not reliable, so you need some pretty sophisticated fuzzy logic functions to make the news useful.
    my bias 2 cents
    peter
    Peter, The fuzzy logic is exactly what CEP companies specialise in, if everyone used a feed of 100% reliable data then everyone would trade in the same way. The point is simply to have access to more information and to derive some logic (fuzzy or solid) that gives you the edge over someone else. -John-
  9. The risk with using tweets is the data is not reliable, so you need some pretty sophisticated fuzzy logic functions to make the news useful.
    my bias 2 cents
    peter

    Peter,
    The fuzzy logic is exactly what CEP companies specialise in, if everyone used a feed of 100% reliable data then everyone would trade in the same way. The point is simply to have access to more information and to derive some logic (fuzzy or solid) that gives you the edge over someone else.

    -John-
    If you look at what many CEP engines offer to day, most do not offer fuzzy logic out of the box. Look at Tim Bass' blog about CEP and you'll see that the majority of them do not handle fuzzy logic. his blog is here. http://www.thecepblog.com/ peter
  10. Look at Tim Bass' blog about CEP and you'll see that the majority of them do not handle fuzzy logic. his blog is here.
    http://www.thecepblog.com/

    peter
    Thanks I'll take a look, off for some lunch :-) -John-
  11. The risk with using tweets is the data is not reliable, so you need some pretty sophisticated fuzzy logic functions to make the news useful.
    my bias 2 cents
    peter

    Peter,
    The fuzzy logic is exactly what CEP companies specialise in, if everyone used a feed of 100% reliable data then everyone would trade in the same way. The point is simply to have access to more information and to derive some logic (fuzzy or solid) that gives you the edge over someone else.

    -John-
    Hi John, I was surprised to read such misinformation in your blog. First of all, you say "The fuzzy logic is exactly what CEP companies specialise in." How can you publish this? There are no companies in the CEP space that specialize in fuzzy logic. To process text required very sophisticated techniques, not query-based rule-style processing. If it was easy to build systems that could process text from unreliable sources, then folks would have done this years ago. I am really surprised to read such misinformation, quite frankly. More surprising is that this is a popular blog and so many people have responded, demonstrating what a wacky world we live in. Yours, Tim
  12. Hi John,

    I was surprised to read such misinformation in your blog.

    First of all, you say "The fuzzy logic is exactly what CEP companies specialise in."

    How can you publish this? There are no companies in the CEP space that specialize in fuzzy logic.

    To process text required very sophisticated techniques, not query-based rule-style processing. If it was easy to build systems that could process text from unreliable sources, then folks would have done this years ago.

    I am really surprised to read such misinformation, quite frankly. More surprising is that this is a popular blog and so many people have responded, demonstrating what a wacky world we live in.

    Yours, Tim
    Hi Tim, Sorry you seem to think this is misinformation, rules used by traders include "fuzzy" logic, my use of the term fuzzy is anything that isn't exact. Having said that, logic can be defined so it's exact, therefore it can't be fuzzy, a contradiction I know. What I meant was that it is quite feasible for a trader to define a rule where... ORCL price greater than 1% above 15 minute moving average and NASDAQ index steady and JAVA (that's Sun) below 1% of opening price and Twitter count of Oracle from TwitterNewsFeeds double usual volume then buy ORCL Something like that, my use of the term fuzzy is the "Twitter count of Oracle from TwitterNewsFeeds double usual volume", it's not an exact science at this point. There's no exact science in trading, if there was there'd be no point in trading. You might feel this is a waste of TSS space but Streambase is written in Java, many of our clients use Java (as well as our C and .NET engines), I'm a technologist and I though it was all pretty damn cool because I was speaking to clients who regularly trade billions of dollars (and Stirling, Yen, Euro, Oil, Gold etc.) and they're interested in this too. My interest is in the technology behind it. -John-
  13. I see a lot of problems with this. First and foremost, this kind of service seems ready-made for market manipulation schemes. Secondly, isn't Twitter famously prone to being overwhelmed? And lastly, if this financial news is so important, why would someone expose it in real time instead of taking advantage of it? They aren't getting paid to do so. Are they just stupid? Or are they selfless saps that want to help other people get/stay rich? On a side note, what about Twitter is special? I don't see much about their model that can't be stolen and improved upon. The only thing that they have is membership and the day that someone provides aggregation for this type of service, that's not going to matter.
  14. I see a lot of problems with this. First and foremost, this kind of service seems ready-made for market manipulation schemes. Secondly, isn't Twitter famously prone to being overwhelmed? And lastly, if this financial news is so important, why would someone expose it in real time instead of taking advantage of it? They aren't getting paid to do so. Are they just stupid? Or are they selfless saps that want to help other people get/stay rich?
    On a side note, what about Twitter is special? I don't see much about their model that can't be stolen and improved upon. The only thing that they have is membership and the day that someone provides aggregation for this type of service, that's not going to matter.
    James, With all due respect I think you're missing the point. Trades are essentially gambling on the way a market will go, up, down etc., much the same way someone betting on a horse to win. People who've done good research tend to have the edge on someone who just likes the name of the horse, someone who knows the trainer and the jockey have an even better edge, any more and you're getting into illegal betting or insider trading as it's known in the trading world. So, if you can improve the source of information then you have an edge over the other traders. Take the example of a Reuters feed announcing a missing plane (not the nicest example I know, sorry), the fact is the plane's missing, nothing else is known, you can speculate on that news and relate it to travel shares. Now you see some noise on Twitter about the airline or the fact that it landed safely on a beach (I was looking for a happy ending), news agencies like Reuters won't announce this until it's conformed but you'll get some noise on Twitter some time before and you can use that to trade your shares before the "official" announcement comes out. What inspired me to "blog" about this on TSS was the news of MJ's death, I first heard about it on Twitter some time before the BBC (my reliable choice of news feed) or CNN any other "reliable" feed broke the news. There was also interesting (sad again) speculation about Jeff Goldblum having died in New Zealand on Twitter, now whether that happened of not (and fortunately it seems like it didn't) people might have started trading the shares of the investment company sponsoring his latest film, if I was a trader (and I'm not), I'd want the opportunity to make money on such things. Streambase's ability to use Twitter as a source of "fuzzy" information would increase my edge on the rest of the market. What makes Twitter so special, nothing, you could set up the same thing, problem is that Twitter's the one people use and you're unlikely to get anywhere with your attempt. If you think you can improve on it then good luck, sell me on the idea and I'll invest some money. -John-
  15. I see a lot of problems with this. First and foremost, this kind of service seems ready-made for market manipulation schemes. Secondly, isn't Twitter famously prone to being overwhelmed? And lastly, if this financial news is so important, why would someone expose it in real time instead of taking advantage of it? They aren't getting paid to do so. Are they just stupid? Or are they selfless saps that want to help other people get/stay rich?
    On a side note, what about Twitter is special? I don't see much about their model that can't be stolen and improved upon. The only thing that they have is membership and the day that someone provides aggregation for this type of service, that's not going to matter.

    James,
    With all due respect I think you're missing the point. Trades are essentially gambling on the way a market will go, up, down etc., much the same way someone betting on a horse to win.
    And that it exactly what's wrong with the stock market. People don't care about owning companies anymore. It's just a big (legal) online casino.
    People who've done good research tend to have the edge on someone who just likes the name of the horse, someone who knows the trainer and the jockey have an even better edge, any more and you're getting into illegal betting or insider trading as it's known in the trading world.

    So, if you can improve the source of information then you have an edge over the other traders. Take the example of a Reuters feed announcing a missing plane (not the nicest example I know, sorry), the fact is the plane's missing, nothing else is known, you can speculate on that news and relate it to travel shares. Now you see some noise on Twitter about the airline or the fact that it landed safely on a beach (I was looking for a happy ending), news agencies like Reuters won't announce this until it's conformed but you'll get some noise on Twitter some time before and you can use that to trade your shares before the "official" announcement comes out.
    What exactly makes you think I don't understand something such painfully simple concept? Don't you see that this can go the other way? Consider this: Six-Year-Old News Story Causes United Airlines Stock to Plummet — UPDATE Google Placed Wrong Date on Story Did the people that dumped their United Airlines stock at nearly $3 (75% loss) benefit from this information? I guess your argument is that the people who got out at the top gained from knowing the false information first. When I see a problem, I don't think, "hey how can I make this problem worse in order to profit" but that's just me. Why should we believe that this kind of thing won't happen with Twitter? How hard would it be to hire a bunch of people to create Twitter accounts and all start 'twittering' about a given stock? Are you not familiar with pump and dump schemes and other similar techniques? We've all seen now that the smart money is pretty stupid. I suggest you read up on the history of the stock market and why news on stocks is so highly regulated.
    What inspired me to "blog" about this on TSS was the news of MJ's death, I first heard about it on Twitter some time before the BBC (my reliable choice of news feed) or CNN any other "reliable" feed broke the news. There was also interesting (sad again) speculation about Jeff Goldblum having died in New Zealand on Twitter, now whether that happened of not (and fortunately it seems like it didn't) people might have started trading the shares of the investment company sponsoring his latest film, if I was a trader (and I'm not), I'd want the opportunity to make money on such things. Streambase's ability to use Twitter as a source of "fuzzy" information would increase my edge on the rest of the market.
    You could also lose money on such things and the reality is that this asinine bullshit has a real effect on companies and the economy. If I had my druthers, taxes on short term gains would be set to excruciatingly costly levels.
    What makes Twitter so special, nothing, you could set up the same thing, problem is that Twitter's the one people use and you're unlikely to get anywhere with your attempt. If you think you can improve on it then good luck, sell me on the idea and I'll invest some money.

    -John-
    Why do people repeat back to me what I just said to them as if they are telling me something I don't know?
  16. What makes Twitter so special, nothing, you could set up the same thing, problem is that Twitter's the one people use and you're unlikely to get anywhere with your attempt.
    Exactly. Just like AIM or AOL in general. Remember when AOL was the a sure bet?
    If you think you can improve on it then good luck, sell me on the idea and I'll invest some money.

    -John-
    I don't really see myself being an outright idea thief. I guess that's a personal flaw. I just don't see how Twitter is ever going to make a profit. Based on your logic Hulu could not manage to come into prominence in the shadow of YouTube. Firefox could never steal IE's market share. The reality is that it's no harder to get 'tweets' off of any other similar service and I doubt it will be long before no one will care about who distributes their 'tweets' any more than we care about how our email gets to it's destination.
  17. What makes Twitter so special, nothing, you could set up the same thing, problem is that Twitter's the one people use and you're unlikely to get anywhere with your attempt.
    Exactly. Just like AIM or AOL in general. Remember when AOL was the a sure bet?
    If you think you can improve on it then good luck, sell me on the idea and I'll invest some money.

    -John-


    I don't really see myself being an outright idea thief. I guess that's a personal flaw. I just don't see how Twitter is ever going to make a profit. Based on your logic Hulu could not manage to come into prominence in the shadow of YouTube. Firefox could never steal IE's market share. The reality is that it's no harder to get 'tweets' off of any other similar service and I doubt it will be long before no one will care about who distributes their 'tweets' any more than we care about how our email gets to it's destination.
    I won't even bother to answer your other post, obviously you have a grudge to bear so I'll leave you with your troubles. I never liked AOL, I don't really care about whether Twitter make money form their business or not as I don't own any equity but one thing is for sure, they have perceived value and that will probably make them a fortune if they sell the company. I wish them luck as I do any entrepreneurs. If you can provide value on top of an already commoditised market place then people will buy your product, one day someone may do that with Tweets, we shall see. -John-
  18. I won't even bother to answer your other post, obviously you have a grudge to bear so I'll leave you with your troubles.
    And ad homimen argument. Instead of responding to a challenge, just attempt to make the challenge appear illegitimate. Is that really the best you can come up with?
  19. I won't even bother to answer your other post, obviously you have a grudge to bear so I'll leave you with your troubles.


    And ad homimen argument. Instead of responding to a challenge, just attempt to make the challenge appear illegitimate. Is that really the best you can come up with?
    My intention was to talk about clever technology not your rants about the ethics of trading. -John-
  20. I won't even bother to answer your other post, obviously you have a grudge to bear so I'll leave you with your troubles.


    And ad homimen argument. Instead of responding to a challenge, just attempt to make the challenge appear illegitimate. Is that really the best you can come up with?

    My intention was to talk about clever technology not your rants about the ethics of trading.

    -John-
    I was talking about the ethics of the clever technology you are proposing and how it appears to attempt to circumvent the regulations you now claim to be keen on.
  21. I never liked AOL, I don't really care about whether Twitter make money form their business or not as I don't own any equity but one thing is for sure, they have perceived value and that will probably make them a fortune if they sell the company.
    And isn't that what it's all these days? Value is not important, it's perceived value. For example, a person can be a complete blowhard and not know what they are talking about but if he or she sounds like they know what they are talking about, that person is considered an expert. pets.com had a lot of 'perceived value' once upon a time. A lot of people are struggling now because the 'perceived value' of their homes has changed drastically over a short period of time. Sure, if you can pass a cubic zirconium off as a diamond, hooray for you. That doesn't mean I have to pretend it's a diamond, though. I just call em like I see em.
  22. A lot of people are struggling now because the 'perceived value' of their homes has changed drastically over a short period of time.
    It's not the perceived value of their homes it's the perceived ability of the tenant to pay back the loans. If someone with little education, a bad job and a history of debts (i.e. sub-prime) is lent money it doesn't take a rocket scientist to understand they might have a problem paying back the loan. The value given to the house was what the tenant was prepared to pay, since they had little hope of ever paying back the money they were happy to pay any price as it was effectively free money. Only when people began to unravel the sub-prime business did the true value of the properties emerge, that price dropped further when fewer people were prepared to buy. Who's fault was it? Unscrupulous mortgage companies, lack of regulation and erroneous risk ratings. If all this had been in place though the US might have had an equally large problem housing and taking care of the millions of poor people. Give the poor a relatively cheap house and endless credit cards that they never pay back and you don't have to worry about paying their medical bills, feeding or housing them. They continue to buy cars and gas on credit and the car industry is supported by the same sub-prime money. Look at it this way and you can start to see why no one bothered to blow the whistle earlier. Back to Streambase and Twitter, this has nothing to do with the above, this will be my last post on this subject, this is a technology forum and I'd rather talk about the technology behind trading engines, FIX engines and algorithmic trading. -John- (sitting in the very warm sun drinking a nice green tea in London).
  23. Back to Streambase and Twitter, this has nothing to do with the above, this will be my last post on this subject, this is a technology forum and I'd rather talk about the technology behind trading engines, FIX engines and algorithmic trading.

    -John- (sitting in the very warm sun drinking a nice green tea in London).
    If you want to discuss the technological side of CEP, you can look at the CEP forum http://forum.complexevents.com/viewforum.php?f=13. In the past, I worked on a natural-language buisness rule language for pre/post trade compliance. In one of your responses, you mentioned DSL. If you want to continue the discussion, email me at woolfel AT gmail DOT com. I'd be interested to see what kind of DSL you're thinking and compare it to what I've done before. peter
  24. If you want to discuss the technological side of CEP, you can look at the CEP forum http://forum.complexevents.com/viewforum.php?f=13. In the past, I worked on a natural-language buisness rule language for pre/post trade compliance. In one of your responses, you mentioned DSL. If you want to continue the discussion, email me at woolfel AT gmail DOT com. I'd be interested to see what kind of DSL you're thinking and compare it to what I've done before.

    peter Thanks, I've taken a look around the forum and I'll drop you a line some time. My email is John dot Davies at OnixS dot biz. -John-
  25. A lot of people are struggling now because the 'perceived value' of their homes has changed drastically over a short period of time.

    It's not the perceived value of their homes it's the perceived ability of the tenant to pay back the loans. If someone with little education, a bad job and a history of debts (i.e. sub-prime) is lent money it doesn't take a rocket scientist to understand they might have a problem paying back the loan. The value given to the house was what the tenant was prepared to pay, since they had little hope of ever paying back the money they were happy to pay any price as it was effectively free money.
    That demonstrates an incomplete understanding of the issues. No one said to themselves "I want to buy a house and have it go into foreclosure in 5 years." You can't take off with a house in-tow. The idea that large numbers of people got mortgages that they didn't intend to pay off is ludicrous. Having lived in an area where home prices inflated rapidly (and then moving to one where they were depressed before purchasing a home) I can tell you that the perceived value was at the root of the problem. It was a bubble just like all the preceding bubbles. People bought homes they could not afford because they believed they would be able to sell or refinance before their adjustable rates kicked in. The expectation was that the value would rise giving them equity which would allow them to get low fixed rate loans or flip the home and use the equity as a down-payment towards a different home. This worked for a number of years as values doubled twice over the span of 5 years in that area and continued to rise. When the prices started to drop, however, people could no longer refinance or sell because they were underwater in their loans. The no-interest (or low fixed rate interest) period of their loans expired and their payments increased. Part of the ignition was that interest rates started rising and this resulted in many peoples monthly payments increasing drastically. In some cases they more than doubled. I actually know someone who bought a house that he could not afford using an interest-only loan and this was the exact reasoning. It wasn't his idea, this is what the 'experts' were saying was a smart move because "real estate doesn't lose value". Even I, who had never even purchased a home at the time, could see that the prices were unsustainable but the 'perceived value' (a friend once called it pixie-dust money) trumped reason. People told me I was crazy when I moved to one of the most depressed (comparatively) real-estate markets to buy a home. It's insane to think that a town-house a stone's throw from a subway station/bus-stop/congested interstate is worth a nearly a million dollars but people figured that it would be worth 2 million before long. And on a side note, this was not a US only problem. Real-estate prices were inflated in many parts of the world. http://www.economist.com/markets/rankings/displaystory.cfm?story_id=13337869&CFID=68027530&CFTOKEN=28493414
  26. Mark Palmer here, CEO of StreamBase. I wrote a blog post that responded to many of the comments / questions on our blog here , including some examples of use cases that we've seen. But I thought I would follow up more directly.... First and foremost, the misperception that responsible traders are making decisions based on Twitter is one we're trying to fight against. Twitter's a great sentiment / rumor filter - "Trade on rumor, sell on news" is one of the oldest trading mantras in the book, and it's legitimate. The example I chose to illustrate its use is to gauge real-time sentiment feedback for retail - for example, during the launch of the iPhone 3Gs. On the day a product like that is released, Apple's stock is volatile, so Twitter's a great way to get feedback from the masses, and quickly. That's what's special about Twitter - you can get fast and quantitative access to the word on the street. Most algo trading treats sentiment as a trading signal, which is interpreted by a live trader. We don't have customers that fully automate decisions on anything coming via Twitter, but getting a sense of sentiment is another matter. As for the question of "isn't Reuters and other news agencies already working on this?" The answer is yes - in fact, they already provide machine-readable feeds (Dow Jones, for example, calls it "Dow Jones Elementized Newsfeed". ) But Reuters and Dow Jones are different - they don't carry the type of sentiment Twitter can deliver - the iPhone example is one. In the blog post I also spoke of Twitter's ability to give direct access to individuals like Richard Branson, who tweets heavily - that's not stuff you get on Reuters or Dow Jones. Yet another example was about trading currency - sentiment reflected in world events such as what's going on in Iran can impact currency prices. And on and on the examples go. No, we didn't get any special non-public link to Twitter - right now, there are some limitations, and its getting better. But for now, and the type of sentiment usage we're seeing, the performance is good enough. Finally, as for our intent, we simply announced this based on customer demand. Indeed, we've linked to Reuters, Dow Jones, blogs, emails, and other sources of information like this before, but Twitter takes the debate to another level, which only reflects the fact that it's a much more powerful communication medium. Yes, we thought it would be interesting and might spark some debate, but we had no idea how much. That said, I think the debate is a good one. Issues of use, legality, the impact on regulation, privacy, and more - it's all fair, and constructive. - Mark
  27. Mark Palmer here, CEO of StreamBase. I wrote a blog post that responded to many of the comments / questions on our blog here , including some examples of use cases that we've seen. But I thought I would follow up more directly....

    First and foremost, the misperception that responsible traders are making decisions based on Twitter is one we're trying to fight against. Twitter's a great sentiment / rumor filter - "Trade on rumor, sell on news" is one of the oldest trading mantras in the book, and it's legitimate.

    The example I chose to illustrate its use is to gauge real-time sentiment feedback for retail - for example, during the launch of the iPhone 3Gs. On the day a product like that is released, Apple's stock is volatile, so Twitter's a great way to get feedback from the masses, and quickly. That's what's special about Twitter - you can get fast and quantitative access to the word on the street. Most algo trading treats sentiment as a trading signal, which is interpreted by a live trader. We don't have customers that fully automate decisions on anything coming via Twitter, but getting a sense of sentiment is another matter.

    As for the question of "isn't Reuters and other news agencies already working on this?" The answer is yes - in fact, they already provide machine-readable feeds (Dow Jones, for example, calls it "Dow Jones Elementized Newsfeed". ) But Reuters and Dow Jones are different - they don't carry the type of sentiment Twitter can deliver - the iPhone example is one. In the blog post I also spoke of Twitter's ability to give direct access to individuals like Richard Branson, who tweets heavily - that's not stuff you get on Reuters or Dow Jones. Yet another example was about trading currency - sentiment reflected in world events such as what's going on in Iran can impact currency prices. And on and on the examples go.

    No, we didn't get any special non-public link to Twitter - right now, there are some limitations, and its getting better. But for now, and the type of sentiment usage we're seeing, the performance is good enough.

    Finally, as for our intent, we simply announced this based on customer demand. Indeed, we've linked to Reuters, Dow Jones, blogs, emails, and other sources of information like this before, but Twitter takes the debate to another level, which only reflects the fact that it's a much more powerful communication medium. Yes, we thought it would be interesting and might spark some debate, but we had no idea how much. That said, I think the debate is a good one. Issues of use, legality, the impact on regulation, privacy, and more - it's all fair, and constructive.

    - Mark
    I would argue it's this kind of irresponsible activities that led to the disaster we see today. Great to see technology help people do things of questional ethical and moral value. Even better that vendors are encouraging the same type of questionable risk taking. If the finance industry insists on such questionable and dubious practices, then I'm sure the government will gladly regulate them. Just because you can, doesn't mean you should. my bias 2 cents. peter
  28. Trading on market sentiment is not what "led to the disaster we see today" - this kind of technology only increases transparency, which is responsible, not irresponsible. Would you mind expanding on the basis for your opinion that this is either dubious and / or questionable? Details, please. - Mark
  29. Trading on market sentiment is not what "led to the disaster we see today" - this kind of technology only increases transparency, which is responsible, not irresponsible. Would you mind expanding on the basis for your opinion that this is either dubious and / or questionable? Details, please.

    - Mark
    from my experience building trading systems, "some" traders feel it's ok to trade on rumor. in the case of institutional traders, that ends up affecting the price of stock, when the rumor is purely non-sense. Using twitter data to me is the same thing. I'm sure others will disagree. The source of the data is not verified, nor is it reliable. It's irresponsible to trade on unsubstantiated rumor. Is it done today? Sure it's done, but it's wrong. That type of behavior is a symptom of rampant greed. For example, when the government cracked down on some "big" firms in boston between 2002-2005, many people in the industry knew it was going to happen. Even back in 2003, I heard traders asking for the ability to gather data from the internet to get a "jump" on their competitors. Trading on unsubstantiated information to me is reckless, pure and simple. Mutual and retirement funds have to follow 1940Act and other investment regulations, but even they have been hurt by the recent problems in the financial sector. Hedge funds currently aren't regulated. I've heard of customers doing some crazy stuff, which in my mind was questionable. Using twitter doesn't make "it" more transparent. On the contrary, many mutual/retirement funds invest in hedge funds, which do not disclose what they invest in, or how they do it. Just look at the news about Madoff and the funds that ended up getting hurt. When a hedge large fund buys/sells, it "can" cause a chain reaction. Using the excuse "others are trading on rumor" to me is non-sense. That's just rationalization. People pushing technology to enable this type of behavior are complicit in my mind. my bias 4 cents. peter
  30. It's irresponsible to trade on unsubstantiated rumor. Is it done today? Sure it's done, but it's wrong. That type of behavior is a symptom of rampant greed.
    I see a balanced view of traders, and trading. Reckless traders exist - agreed. Trading on unsubstantiated rumors is reckless - agreed. Madoff was a crook - agreed. On the other hand, the vast majority of traders I know are thoughtful, intelligent, and ethical. Both types of traders - reckless and thoughtful - use news and sentiment to inform their decisions. Sometimes, a thoughtful trader will be informed of sentiment, recognize it as folly, and do nothing. That's a constructive use of news, and it's why Bloomberg, CNN, and other news sources sit on trader desks all over the world. It's why Reuters and Dow Jones transmit their news in machine-readable formats. Twitter's is just another input, with different characteristics. Should we take CNN and Bloomberg off trading floors too? - Mark
  31. It's irresponsible to trade on unsubstantiated rumor. Is it done today? Sure it's done, but it's wrong. That type of behavior is a symptom of rampant greed.

    I see a balanced view of traders, and trading. Reckless traders exist - agreed. Trading on unsubstantiated rumors is reckless - agreed. Madoff was a crook - agreed.

    On the other hand, the vast majority of traders I know are thoughtful, intelligent, and ethical.

    Both types of traders - reckless and thoughtful - use news and sentiment to inform their decisions. Sometimes, a thoughtful trader will be informed of sentiment, recognize it as folly, and do nothing. That's a constructive use of news, and it's why Bloomberg, CNN, and other news sources sit on trader desks all over the world. It's why Reuters and Dow Jones transmit their news in machine-readable formats. Twitter's is just another input, with different characteristics. Should we take CNN and Bloomberg off trading floors too?

    - Mark
    I have met good traders and I agree they aren't all bad. Having said that, there are a lot of shady people in the industry. if you know the boston financial sector, you'll know what happened at the top 5 firms between 2003 and 2007. Since I worked on pre and post trade compliance, I saw a lot of questionable behavior, which trade side people never see or hear about. peter
  32. I would argue it's this kind of irresponsible activities that led to the disaster we see today. Great to see technology help people do things of questional ethical and moral value. Even better that vendors are encouraging the same type of questionable risk taking.

    If the finance industry insists on such questionable and dubious practices, then I'm sure the government will gladly regulate them. Just because you can, doesn't mean you should.
    my bias 2 cents.
    peter
    Peter, Trading and making a profit is a fact of human existence, you need to make a profit to survive over the lean times. If a broker or trader didn't make any money then you wouldn't get interest on your pension, banks wouldn't have money to lend so that you can buy houses and cars and industries wouldn't have financing so invest in R&D to develop cool gadgets like the iPhone. That's just the way it is, it's not all ethical I agree and I hope we see more clamp-downs on the less ethical but like it or not Twitter is useful to traders and I can't see that comparing to the likes of Madoff. -John-
  33. I would argue it's this kind of irresponsible activities that led to the disaster we see today. Great to see technology help people do things of questional ethical and moral value. Even better that vendors are encouraging the same type of questionable risk taking.

    If the finance industry insists on such questionable and dubious practices, then I'm sure the government will gladly regulate them. Just because you can, doesn't mean you should.
    my bias 2 cents.
    peter

    Peter,
    Trading and making a profit is a fact of human existence, you need to make a profit to survive over the lean times. If a broker or trader didn't make any money then you wouldn't get interest on your pension, banks wouldn't have money to lend so that you can buy houses and cars and industries wouldn't have financing so invest in R&D to develop cool gadgets like the iPhone. That's just the way it is, it's not all ethical I agree and I hope we see more clamp-downs on the less ethical but like it or not Twitter is useful to traders and I can't see that comparing to the likes of Madoff.

    -John-
    I agree trading is part of life. I'm fine with ethical trading. I agree twitter data "can" be useful. The real question for me is "do these tools really deliver value?" to me, it's mostly hype. I've been studying AI for 8 years now. Fuzzy logic and machine learning are very difficult to implement. I have serious doubts the current CEP market leaders can handle fuzzy logic and machine learning. Most of them are stream processing engines and that's it. They aren't designed for AI. peter
  34. I agree trading is part of life. I'm fine with ethical trading. I agree twitter data "can" be useful. The real question for me is "do these tools really deliver value?" to me, it's mostly hype. I've been studying AI for 8 years now. Fuzzy logic and machine learning are very difficult to implement. I have serious doubts the current CEP market leaders can handle fuzzy logic and machine learning. Most of them are stream processing engines and that's it. They aren't designed for AI.

    peter
    Do they deliver value? Well you'll have to talk to the StreamBase guys about that. I'd guess like any small to medium sized company they're more into providing what customers what than hype, the hype tends to come from the larger companies that don't actually have product or the agility to move their product into the right market so they create hype around what they do do like "SOA" and other related crap. It's interesting to see companies like Microsoft, IBM and RedHat getting into CEP with what's little more than a rules engine. Of course their marketing people will tell you it's the latest and greatest but you can be sure it wasn't designed to do what it's doing today or as in the case of the first two, only runs on their propitiatory OS or kit. CEP is not necessarily AI, it's not a rules engine either, they are all different things. The origin of CEP tends to be, in my own opinion, from algorithmic trading and unless it's from that domain I really don't think you can fit a rules engine into the CEP category. AI on the other hand has a fairly long relationship with trading so I guess there's an element of overlap. Given that a good CEP product is really just a application with programming tools then AI could/would be part of that. A lot of the work I do for clients involves very high performance (and volume) processing so I come across the need for rules engines and CEP pretty frequently. AI is a big part of that but what I look for in an engine (CEP or rules) is a flexible DSL or language that is easy to recruit for and obviously a scalable architecture. -John-
  35. I agree trading is part of life. I'm fine with ethical trading. I agree twitter data "can" be useful. The real question for me is "do these tools really deliver value?" to me, it's mostly hype. I've been studying AI for 8 years now. Fuzzy logic and machine learning are very difficult to implement. I have serious doubts the current CEP market leaders can handle fuzzy logic and machine learning. Most of them are stream processing engines and that's it. They aren't designed for AI.

    peter


    Do they deliver value? Well you'll have to talk to the StreamBase guys about that. I'd guess like any small to medium sized company they're more into providing what customers what than hype, the hype tends to come from the larger companies that don't actually have product or the agility to move their product into the right market so they create hype around what they do do like "SOA" and other related crap.

    It's interesting to see companies like Microsoft, IBM and RedHat getting into CEP with what's little more than a rules engine. Of course their marketing people will tell you it's the latest and greatest but you can be sure it wasn't designed to do what it's doing today or as in the case of the first two, only runs on their propitiatory OS or kit.

    CEP is not necessarily AI, it's not a rules engine either, they are all different things. The origin of CEP tends to be, in my own opinion, from algorithmic trading and unless it's from that domain I really don't think you can fit a rules engine into the CEP category. AI on the other hand has a fairly long relationship with trading so I guess there's an element of overlap. Given that a good CEP product is really just a application with programming tools then AI could/would be part of that.

    A lot of the work I do for clients involves very high performance (and volume) processing so I come across the need for rules engines and CEP pretty frequently. AI is a big part of that but what I look for in an engine (CEP or rules) is a flexible DSL or language that is easy to recruit for and obviously a scalable architecture.

    -John-
    I agree a CEP engine is different than a general purpose business rule engine, which is also different than AI. My point is that using twitter feeds with the existing CEP engines is impractical, since they do not support fuzzy logic. I've dabbled with FuzzyJess and studied fuzzy logic. I'm not an expert or intermediate user of fuzzy logic. What I do know is that it's very hard to implement effectively and isn't something an average developer can learn in a few months or even 1-2 years. it would be trivial for a person to use zombie systems to generate tweet traffic about a company or stock and start a chain reaction. My fear irrational as it may be is that once a trader starts to use twitter data, someone else will exploit that. A lot of by standers may end up getting hurt by inappropriate use of technology. for the record, I feel twitter-like data may be useful in the future, but right now the noise is far too high. peter
  36. I would argue it's this kind of irresponsible activities that led to the disaster we see today. Great to see technology help people do things of questional ethical and moral value. Even better that vendors are encouraging the same type of questionable risk taking.

    If the finance industry insists on such questionable and dubious practices, then I'm sure the government will gladly regulate them. Just because you can, doesn't mean you should.
    my bias 2 cents.
    peter

    Peter,
    Trading and making a profit is a fact of human existence, you need to make a profit to survive over the lean times. If a broker or trader didn't make any money then you wouldn't get interest on your pension, banks wouldn't have money to lend so that you can buy houses and cars and industries wouldn't have financing so invest in R&D to develop cool gadgets like the iPhone. That's just the way it is, it's not all ethical I agree and I hope we see more clamp-downs on the less ethical but like it or not Twitter is useful to traders and I can't see that comparing to the likes of Madoff.


    -John-


    I agree trading is part of life. I'm fine with ethical trading. I agree twitter data "can" be useful. The real question for me is "do these tools really deliver value?" to me, it's mostly hype. I've been studying AI for 8 years now. Fuzzy logic and machine learning are very difficult to implement. I have serious doubts the current CEP market leaders can handle fuzzy logic and machine learning. Most of them are stream processing engines and that's it. They aren't designed for AI.

    peter
    I agree with Peter. Folks get both emotional and excited about technology, blinding them to actually understanding what they are talking about. That's OK, but there is also humor in it. Seriously, It takes very sophisticated processing to take textual data from myriad untrusted sources and determine anything of value. AI folks have struggled with this for decades, as Peter points out. Governments have struggled with this for decades in their intelligence processing systems. So, being people of logic and human intelligence, how can query-based stream processors that has no sophisticated logic (AI, fuzzy, Baysian, neural or otherwise) compared to decades of technology development and implementation by some of the world's best and brightest engineers and scientist accomplish what no system has reliably accomplished before? (Include Star Trek Theme Music).... The short answer: It can't. Anyone who believes that simple query-oriented stream processing engines can process twitter tweeks, which can (must) be modeled as an unreliable, untrusted information channel with a very low signal to noise ratio, does not understand the math and science behind signal processing and related science. IF twitter was from trusted sources and IF there was an agree upon semantic structure, and IF the channel was reliable and IF ... IF ... IF .... As I said in my comment to one of Streambase's evangelists: “Announcement: Streambase has processing intelligence that no other company in the world has, no matter how great or big. We have the “Holy Grail Stream Processing Engine. You can buy it, plug it in, process Twitter rumors, and never have to work another day in your life! From this post and comments: http://www.thecepblog.com/2009/06/29/gimmick-marketing-why/ Cheers, Tim
  37. nice[ Go to top ]

    Very nice. I'd think this is a hell lot more realistic than some quant guy coming up with a random quant signal and pointing to it and declare that it represents consumer sentiment and can forecast off of it.
  38. Of course Twitter as a information source is a good idea. No different than having CNN on the trader's desk. Let's get off the "traders are evil" rants and back to technology.... What kind of connectivity does Twitter allow and how do you link it to StreamBase? What kind of data rates can you push through? John, do you use Twitter and how? - Jim
  39. Of course Twitter as a information source is a good idea. No different than having CNN on the trader's desk. Let's get off the "traders are evil" rants and back to technology....

    What kind of connectivity does Twitter allow and how do you link it to StreamBase? What kind of data rates can you push through? John, do you use Twitter and how?

    - Jim
    Personally I use Twitter yes, I don't trade though and I don't work for or with Streambase so you'll have to wait for a Streambase authority to answer detailed questions. I work in the trading environment and all of my customers are traders or exchanges so I'm very close to the "coal front" so to speak. I just thought it was an interesting development and thought Streambase were being rather clever and forward thinking with their announcement. -John-
  40. Of course Twitter as a information source is a good idea. No different than having CNN on the trader's desk. Let's get off the "traders are evil" rants and back to technology...
    It's not the same as CNN. That's my point and I never said anyone was evil. I implied that some people that are traders are stupid and I stand by that. Maybe you and John could quit with the rhetoric and try to understand.
  41. Of course Twitter as a information source is a good idea. No different than having CNN on the trader's desk. Let's get off the "traders are evil" rants and back to technology...


    It's not the same as CNN. That's my point and I never said anyone was evil. I implied that some people that are traders are stupid and I stand by that. Maybe you and John could quit with the rhetoric and try to understand.
    Actually, it was me who suggested "some" traders are evil. I stand by that. Not all traders are evil. Back to technology. What most CEP engines do today I classify under decision automation. It uses simplistic rules, with a fixed algorithm for deciding when to trade a security, or how to execute the trade to optimize the cost. One obvious example is the volume weighted average price, which is only good for executing trades once a human being decides to purchase X dollars of Y stock. What business rules don't handle well is deciphering what triggered the change. To do that, the system has to have a knowledge base and utilize fuzzy logic to make a "good guess". What some vendors are calling knowledge base today is just a business dictionary. Within AI there's the concept of strong negation, weak negation, unknown, and unknowable. For example, a system that tries to model or predict stocks can't say with 100% certainty why a stock is moving up or down. That means the system has to support the notion of degrees of truth. The rules a developer or business person would write has to be in those terms. It gets even more complex when you take into account the system has to change and adapt on the fly to changing input. If a developer hard codes a function, that probably won't work, since the environment is highly dynamic. In my bias opinion, the current crop of event processing engines are not powerful enough to handle these types of use cases. I hope one day they will. peter
  42. Interesting commentary, the AI stuff (IMHO) is getting a bit too esoteric for my taste. CEP is good at bringing multiple streams of data together to correlate patterns - in this case news and trading. This is a common design pattern we've (at Apama) used for quite come time with Dow Jones New Service. The 'challenge' of doing the same with Twitter is latency and validity. DJ/Reuters are sanctioned news sources focused on low-latency and pride themselves on accuracy in reporting. Neither of these can be applied to information on Twitter. Here's a couple of others making comments on Twitter as a news source: http://www.alacrablog.com/alacrablog/2009/06/twitter-is-the-herd-or-trading-with-twitter.html http://www.marketwatch.com/story/as-news-source-twitter-has-seven-deadly-flaws