667481 members! Sign up to stay informed.

Sponsored Links


Resources

Enterprise Java
Research Library

Get Java white papers, product information, case studies and webcasts

News News News Messages: 43 Messages: 43 Messages: 43 Printer friendly Printer friendly Printer friendly Post reply Post reply Post reply XML XML XML

Algorithmic trading with Twitter

Posted by: John Davies on June 29, 2009 DIGG
A year ago someone asked me if I'd tried Twitter, "why would I want to know what people are watching on TV?" was my response. Just over a month ago I was asked if Twitter was a viable media for updating our clients about product updates, I'm ashamed to admit I said "no". This week I find that traders are using Twitter as a feed for algorithmic trading, are they mad?

My company sell FIX engines specifically tuned to connect to exchanges, places like the CME (Chicago Mercantile Exchange), the ICE (Intercontinental Exchange) and NYSE (New York Stock Exchange), we have 100 odd customers ranging from a couple of guys trading their own money in Monaco to extremely large petrochemical companies in Houston, Texas so it's fare to say we know a thing or two about who's trading what and how.

When I saw a press release this week, as part of a finance show (SIFMA in New York), about StreamBase developing a Twitter interface to their CEP engine I initially thought they were either mad or simply going for consumer press coverage. After talking to a few of our customers it seems as though this is actually critical to today's trading.

For the last 20 plus years we'd think nothing of using news feeds from Reuters as a source of information, if an oil-well was hit by a hurricane, oil prices went up, if a company founder or CEO died the share-price of the company changed drastically, if a riot broke out in a country it could effect the exchange rate of its currency. Basically, once the news is on Reuters it's official and the prices start to change as soon as the news comes out.

If you're going to make money though you need to be the first with the news, if the price is going to fall you want to sell (or borrow if you're shorting) before it falls, not during or afterwards. If it's about to go up you want to buy before it goes up and not after it's too late.

So, how do you know if Twitter coverage is real or just Twitter chatter? Well this is where a CEP (Complex Event Processing) engine comes in. You listen to everything and write clever algorithms (functions or methods) that aggregate the news and decide on it's significance. Once you decide it's probably true then you trade on that information. Bear in mind that other people are also watching the same "news" so they are likely to trade too and even if the news is wrong you still need to be trading before everyone else does.

So it seems Twitter has become a vital part of my FX (Foreign Exchange) and energies trading world, Streambase were first and for that they deserve some qdos, we'll now see the larger, less agile companies following suite, the nicest form of flattery.

Neither I nor my company have any connection with StreamBase, I just thought this might be an interesting subject to discuss.

-John Davies-
Technical Director
Onix
Twitter: @jtdavies

Threaded replies

·  Algorithmic trading with Twitter by John Davies on Mon Jun 29 08:08:18 EDT 2009
  ·  Re: Algorithmic trading with Twitter by Benoit Xhenseval on Mon Jun 29 11:57:48 EDT 2009
    ·  Re: Algorithmic trading with Twitter by John Davies on Tue Jun 30 08:08:26 EDT 2009
    ·  Re: Algorithmic trading with Twitter by John Davies on Tue Jun 30 08:08:27 EDT 2009
      ·  Re: Algorithmic trading with Twitter by Benoit Xhenseval on Tue Jun 30 10:29:22 EDT 2009
    ·  On a more humurous note, even the dead can tweet by peter lin on Tue Jun 30 12:29:42 EDT 2009
  ·  News agencies already provide this kind of service by peter lin on Mon Jun 29 12:07:36 EDT 2009
    ·  Re: News agencies already provide this kind of service by John Davies on Tue Jun 30 08:11:42 EDT 2009
      ·  Re: News agencies already provide this kind of service by peter lin on Tue Jun 30 08:33:04 EDT 2009
        ·  Re: News agencies already provide this kind of service by John Davies on Tue Jun 30 08:40:05 EDT 2009
      ·  Re: News agencies already provide this kind of service by Tim Bass on Thu Jul 02 06:24:25 EDT 2009
        ·  Re: News agencies already provide this kind of service by John Davies on Thu Jul 02 09:30:29 EDT 2009
  ·  Re: Algorithmic trading with Twitter by James Watson on Mon Jun 29 17:18:44 EDT 2009
    ·  Re: Algorithmic trading with Twitter by John Davies on Tue Jun 30 08:27:31 EDT 2009
      ·  Re: Algorithmic trading with Twitter by James Watson on Tue Jun 30 12:22:29 EDT 2009
      ·  Re: Algorithmic trading with Twitter by James Watson on Tue Jun 30 12:46:31 EDT 2009
        ·  Re: Algorithmic trading with Twitter by John Davies on Tue Jun 30 16:50:37 EDT 2009
          ·  Re: Algorithmic trading with Twitter by James Watson on Tue Jun 30 17:25:21 EDT 2009
            ·  Re: Algorithmic trading with Twitter by John Davies on Wed Jul 01 04:04:08 EDT 2009
              ·  Re: Algorithmic trading with Twitter by James Watson on Wed Jul 01 11:25:20 EDT 2009
          ·  Re: Algorithmic trading with Twitter by James Watson on Tue Jun 30 17:55:08 EDT 2009
            ·  Re: Algorithmic trading with Twitter by John Davies on Wed Jul 01 04:45:38 EDT 2009
              ·  Re: Algorithmic trading with Twitter by peter lin on Wed Jul 01 07:50:50 EDT 2009
                ·  Re: Algorithmic trading with Twitter by John Davies on Wed Jul 01 17:25:43 EDT 2009
              ·  Re: Algorithmic trading with Twitter by James Watson on Wed Jul 01 11:05:45 EDT 2009
  ·  Comments on Twitter, from StreamBase by Mark Palmer on Mon Jun 29 18:37:41 EDT 2009
    ·  Re: Comments on Twitter, from StreamBase by peter lin on Mon Jun 29 19:22:24 EDT 2009
      ·  Substantiate: "irresponsible," and "what led to the disaster" by Mark Palmer on Mon Jun 29 20:31:08 EDT 2009
        ·  Re: Substantiate: "irresponsible," and "what led to the disaster" by peter lin on Mon Jun 29 21:44:49 EDT 2009
          ·  Re: Substantiate: "irresponsible," and "what led to the disaster by Mark Palmer on Tue Jun 30 01:11:08 EDT 2009
            ·  Re: Substantiate: "irresponsible," and "what led to the disaster by peter lin on Tue Jun 30 08:37:17 EDT 2009
      ·  Re: Comments on Twitter, from StreamBase by John Davies on Tue Jun 30 08:38:30 EDT 2009
        ·  Re: Comments on Twitter, from StreamBase by peter lin on Tue Jun 30 09:38:01 EDT 2009
          ·  Re: Comments on Twitter, from StreamBase by John Davies on Tue Jun 30 10:37:26 EDT 2009
            ·  Re: Comments on Twitter, from StreamBase by peter lin on Tue Jun 30 11:10:37 EDT 2009
          ·  Re: Comments on Twitter, from StreamBase by Tim Bass on Thu Jul 02 06:46:00 EDT 2009
    ·  nice by Steven Peh on Mon Jun 29 21:32:21 EDT 2009
  ·  GOOD by jock guck on Wed Jul 01 01:16:48 EDT 2009
    ·  How to use rel=nofollow attribute to discourage SEO spammers by Casual Visitor on Wed Jul 01 15:48:46 EDT 2009
  ·  Of course this is happening - technical detail? by Jim Walsh on Thu Jul 02 01:57:52 EDT 2009
    ·  Re: Of course this is happening - technical detail? by John Davies on Thu Jul 02 09:15:30 EDT 2009
    ·  Re: Of course this is happening - technical detail? by James Watson on Thu Jul 02 09:32:46 EDT 2009
      ·  Re: Of course this is happening - technical detail? by peter lin on Thu Jul 02 14:18:54 EDT 2009
  ·  Re: Algorithmic trading with Twitter by Louie Lovas on Thu Jul 02 11:43:26 EDT 2009
  Message #310401 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: Benoit Xhenseval on June 29, 2009 in response to Message #310348
Very interesting...

I've attended 2 Devnest conferences in the City of London (http://twitter.com/devnest) and quite frankly, I was the only developer from the financial industry... So, i was wondering if I was mad to think about this...

One thing that I have learned however is that only a very limited number of "trusted" companies have access to the FULL feed from Twitter (hose-pipe). And I believe these to be the like of Google, Bing (very very limited). Everyone else has access to a limited "selected" feed, it is possible to request a better feed but the whole process is opaque and based on who you know more than anything else (these are my views btw).

So, did StreamBase manage to get access to the firehose? I doubt... so the StreamBase algo will make decisions based on a limited set of information... oh, hang on, that is exactly like trading...

Very interesting, I wonder what kind of Tweet they push out... (spamming to create a buzz?)

Our company is using Twitter but simply to post some server status info... still very low level, I know.

Interesting stuff,

Benoit
Http://www.Appendium.com

  Message #310403 Post reply Post reply Post reply Go to top Go to top Go to top

News agencies already provide this kind of service

Posted by: peter lin on June 29, 2009 in response to Message #310348
I think reuters and other news agencies were already working on these types of services. Not sure if anyone is using it or how. The risk with using tweets is the data is not reliable, so you need some pretty sophisticated fuzzy logic functions to make the news useful. To the best of my knowledge, none of the commercial products have built-in fuzzy logic or real inference capabilities. Whether it's a gimmick or practical in "real life" is yet to be seen.

my bias 2 cents

peter

  Message #310418 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: James Watson on June 29, 2009 in response to Message #310348
I see a lot of problems with this. First and foremost, this kind of service seems ready-made for market manipulation schemes. Secondly, isn't Twitter famously prone to being overwhelmed? And lastly, if this financial news is so important, why would someone expose it in real time instead of taking advantage of it? They aren't getting paid to do so. Are they just stupid? Or are they selfless saps that want to help other people get/stay rich?

On a side note, what about Twitter is special? I don't see much about their model that can't be stolen and improved upon. The only thing that they have is membership and the day that someone provides aggregation for this type of service, that's not going to matter.

  Message #310426 Post reply Post reply Post reply Go to top Go to top Go to top

Comments on Twitter, from StreamBase

Posted by: Mark Palmer on June 29, 2009 in response to Message #310348
Mark Palmer here, CEO of StreamBase. I wrote a blog post that responded to many of the comments / questions on our blog here , including some examples of use cases that we've seen. But I thought I would follow up more directly....

First and foremost, the misperception that responsible traders are making decisions based on Twitter is one we're trying to fight against. Twitter's a great sentiment / rumor filter - "Trade on rumor, sell on news" is one of the oldest trading mantras in the book, and it's legitimate.

The example I chose to illustrate its use is to gauge real-time sentiment feedback for retail - for example, during the launch of the iPhone 3Gs. On the day a product like that is released, Apple's stock is volatile, so Twitter's a great way to get feedback from the masses, and quickly. That's what's special about Twitter - you can get fast and quantitative access to the word on the street. Most algo trading treats sentiment as a trading signal, which is interpreted by a live trader. We don't have customers that fully automate decisions on anything coming via Twitter, but getting a sense of sentiment is another matter.

As for the question of "isn't Reuters and other news agencies already working on this?" The answer is yes - in fact, they already provide machine-readable feeds (Dow Jones, for example, calls it "Dow Jones Elementized Newsfeed". ) But Reuters and Dow Jones are different - they don't carry the type of sentiment Twitter can deliver - the iPhone example is one. In the blog post I also spoke of Twitter's ability to give direct access to individuals like Richard Branson, who tweets heavily - that's not stuff you get on Reuters or Dow Jones. Yet another example was about trading currency - sentiment reflected in world events such as what's going on in Iran can impact currency prices. And on and on the examples go.

No, we didn't get any special non-public link to Twitter - right now, there are some limitations, and its getting better. But for now, and the type of sentiment usage we're seeing, the performance is good enough.

Finally, as for our intent, we simply announced this based on customer demand. Indeed, we've linked to Reuters, Dow Jones, blogs, emails, and other sources of information like this before, but Twitter takes the debate to another level, which only reflects the fact that it's a much more powerful communication medium. Yes, we thought it would be interesting and might spark some debate, but we had no idea how much. That said, I think the debate is a good one. Issues of use, legality, the impact on regulation, privacy, and more - it's all fair, and constructive.

- Mark

  Message #310427 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Comments on Twitter, from StreamBase

Posted by: peter lin on June 29, 2009 in response to Message #310426
Mark Palmer here, CEO of StreamBase. I wrote a blog post that responded to many of the comments / questions on our blog here , including some examples of use cases that we've seen. But I thought I would follow up more directly....

First and foremost, the misperception that responsible traders are making decisions based on Twitter is one we're trying to fight against. Twitter's a great sentiment / rumor filter - "Trade on rumor, sell on news" is one of the oldest trading mantras in the book, and it's legitimate.

The example I chose to illustrate its use is to gauge real-time sentiment feedback for retail - for example, during the launch of the iPhone 3Gs. On the day a product like that is released, Apple's stock is volatile, so Twitter's a great way to get feedback from the masses, and quickly. That's what's special about Twitter - you can get fast and quantitative access to the word on the street. Most algo trading treats sentiment as a trading signal, which is interpreted by a live trader. We don't have customers that fully automate decisions on anything coming via Twitter, but getting a sense of sentiment is another matter.

As for the question of "isn't Reuters and other news agencies already working on this?" The answer is yes - in fact, they already provide machine-readable feeds (Dow Jones, for example, calls it "Dow Jones Elementized Newsfeed". ) But Reuters and Dow Jones are different - they don't carry the type of sentiment Twitter can deliver - the iPhone example is one. In the blog post I also spoke of Twitter's ability to give direct access to individuals like Richard Branson, who tweets heavily - that's not stuff you get on Reuters or Dow Jones. Yet another example was about trading currency - sentiment reflected in world events such as what's going on in Iran can impact currency prices. And on and on the examples go.

No, we didn't get any special non-public link to Twitter - right now, there are some limitations, and its getting better. But for now, and the type of sentiment usage we're seeing, the performance is good enough.

Finally, as for our intent, we simply announced this based on customer demand. Indeed, we've linked to Reuters, Dow Jones, blogs, emails, and other sources of information like this before, but Twitter takes the debate to another level, which only reflects the fact that it's a much more powerful communication medium. Yes, we thought it would be interesting and might spark some debate, but we had no idea how much. That said, I think the debate is a good one. Issues of use, legality, the impact on regulation, privacy, and more - it's all fair, and constructive.

- Mark


I would argue it's this kind of irresponsible activities that led to the disaster we see today. Great to see technology help people do things of questional ethical and moral value. Even better that vendors are encouraging the same type of questionable risk taking.

If the finance industry insists on such questionable and dubious practices, then I'm sure the government will gladly regulate them. Just because you can, doesn't mean you should.

my bias 2 cents.

peter

  Message #310428 Post reply Post reply Post reply Go to top Go to top Go to top

Substantiate: "irresponsible," and "what led to the disaster"

Posted by: Mark Palmer on June 29, 2009 in response to Message #310427
Trading on market sentiment is not what "led to the disaster we see today" - this kind of technology only increases transparency, which is responsible, not irresponsible. Would you mind expanding on the basis for your opinion that this is either dubious and / or questionable? Details, please.

- Mark

  Message #310429 Post reply Post reply Post reply Go to top Go to top Go to top

nice

Posted by: Steven Peh on June 29, 2009 in response to Message #310426
Very nice. I'd think this is a hell lot more realistic than some quant guy coming up with a random quant signal and pointing to it and declare that it represents consumer sentiment and can forecast off of it.

  Message #310430 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Substantiate: "irresponsible," and "what led to the disaster"

Posted by: peter lin on June 29, 2009 in response to Message #310428
Trading on market sentiment is not what "led to the disaster we see today" - this kind of technology only increases transparency, which is responsible, not irresponsible. Would you mind expanding on the basis for your opinion that this is either dubious and / or questionable? Details, please.

- Mark


from my experience building trading systems, "some" traders feel it's ok to trade on rumor. in the case of institutional traders, that ends up affecting the price of stock, when the rumor is purely non-sense. Using twitter data to me is the same thing. I'm sure others will disagree. The source of the data is not verified, nor is it reliable. It's irresponsible to trade on unsubstantiated rumor. Is it done today? Sure it's done, but it's wrong. That type of behavior is a symptom of rampant greed.

For example, when the government cracked down on some "big" firms in boston between 2002-2005, many people in the industry knew it was going to happen. Even back in 2003, I heard traders asking for the ability to gather data from the internet to get a "jump" on their competitors. Trading on unsubstantiated information to me is reckless, pure and simple.

Mutual and retirement funds have to follow 1940Act and other investment regulations, but even they have been hurt by the recent problems in the financial sector. Hedge funds currently aren't regulated. I've heard of customers doing some crazy stuff, which in my mind was questionable.

Using twitter doesn't make "it" more transparent. On the contrary, many mutual/retirement funds invest in hedge funds, which do not disclose what they invest in, or how they do it. Just look at the news about Madoff and the funds that ended up getting hurt. When a hedge large fund buys/sells, it "can" cause a chain reaction. Using the excuse "others are trading on rumor" to me is non-sense. That's just rationalization. People pushing technology to enable this type of behavior are complicit in my mind.

my bias 4 cents.

peter

  Message #310438 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Substantiate: "irresponsible," and "what led to the disaster

Posted by: Mark Palmer on June 30, 2009 in response to Message #310430
It's irresponsible to trade on unsubstantiated rumor. Is it done today? Sure it's done, but it's wrong. That type of behavior is a symptom of rampant greed.

I see a balanced view of traders, and trading. Reckless traders exist - agreed. Trading on unsubstantiated rumors is reckless - agreed. Madoff was a crook - agreed.

On the other hand, the vast majority of traders I know are thoughtful, intelligent, and ethical.

Both types of traders - reckless and thoughtful - use news and sentiment to inform their decisions. Sometimes, a thoughtful trader will be informed of sentiment, recognize it as folly, and do nothing. That's a constructive use of news, and it's why Bloomberg, CNN, and other news sources sit on trader desks all over the world. It's why Reuters and Dow Jones transmit their news in machine-readable formats. Twitter's is just another input, with different characteristics. Should we take CNN and Bloomberg off trading floors too?

- Mark

  Message #310482 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: John Davies on June 30, 2009 in response to Message #310401
Benoit,
I'm not sure you'd actually need access to the "firehose" as you put it, you just need a feed to the usual news sources and perhaps a few others and you let the clever logic sort out whether something is worth acting on or not. Of course access to the full "firehose" would be interesting but it's probably a lot more noise than you really need.

-John-

  Message #310483 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: John Davies on June 30, 2009 in response to Message #310401
Benoit,
I'm not sure you'd actually need access to the "firehose" as you put it, you just need a feed to the usual news sources and perhaps a few others and you let the clever logic sort out whether something is worth acting on or not. Of course access to the full "firehose" would be interesting but it's probably a lot more noise than you really need.

-John-

  Message #310484 Post reply Post reply Post reply Go to top Go to top Go to top

Re: News agencies already provide this kind of service

Posted by: John Davies on June 30, 2009 in response to Message #310403
The risk with using tweets is the data is not reliable, so you need some pretty sophisticated fuzzy logic functions to make the news useful.
my bias 2 cents
peter

Peter,
The fuzzy logic is exactly what CEP companies specialise in, if everyone used a feed of 100% reliable data then everyone would trade in the same way. The point is simply to have access to more information and to derive some logic (fuzzy or solid) that gives you the edge over someone else.

-John-

  Message #310486 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: John Davies on June 30, 2009 in response to Message #310418
I see a lot of problems with this. First and foremost, this kind of service seems ready-made for market manipulation schemes. Secondly, isn't Twitter famously prone to being overwhelmed? And lastly, if this financial news is so important, why would someone expose it in real time instead of taking advantage of it? They aren't getting paid to do so. Are they just stupid? Or are they selfless saps that want to help other people get/stay rich?
On a side note, what about Twitter is special? I don't see much about their model that can't be stolen and improved upon. The only thing that they have is membership and the day that someone provides aggregation for this type of service, that's not going to matter.

James,
With all due respect I think you're missing the point. Trades are essentially gambling on the way a market will go, up, down etc., much the same way someone betting on a horse to win. People who've done good research tend to have the edge on someone who just likes the name of the horse, someone who knows the trainer and the jockey have an even better edge, any more and you're getting into illegal betting or insider trading as it's known in the trading world.

So, if you can improve the source of information then you have an edge over the other traders. Take the example of a Reuters feed announcing a missing plane (not the nicest example I know, sorry), the fact is the plane's missing, nothing else is known, you can speculate on that news and relate it to travel shares. Now you see some noise on Twitter about the airline or the fact that it landed safely on a beach (I was looking for a happy ending), news agencies like Reuters won't announce this until it's conformed but you'll get some noise on Twitter some time before and you can use that to trade your shares before the "official" announcement comes out.

What inspired me to "blog" about this on TSS was the news of MJ's death, I first heard about it on Twitter some time before the BBC (my reliable choice of news feed) or CNN any other "reliable" feed broke the news. There was also interesting (sad again) speculation about Jeff Goldblum having died in New Zealand on Twitter, now whether that happened of not (and fortunately it seems like it didn't) people might have started trading the shares of the investment company sponsoring his latest film, if I was a trader (and I'm not), I'd want the opportunity to make money on such things. Streambase's ability to use Twitter as a source of "fuzzy" information would increase my edge on the rest of the market.

What makes Twitter so special, nothing, you could set up the same thing, problem is that Twitter's the one people use and you're unlikely to get anywhere with your attempt. If you think you can improve on it then good luck, sell me on the idea and I'll invest some money.

-John-

  Message #310487 Post reply Post reply Post reply Go to top Go to top Go to top

Re: News agencies already provide this kind of service

Posted by: peter lin on June 30, 2009 in response to Message #310484
The risk with using tweets is the data is not reliable, so you need some pretty sophisticated fuzzy logic functions to make the news useful.
my bias 2 cents
peter

Peter,
The fuzzy logic is exactly what CEP companies specialise in, if everyone used a feed of 100% reliable data then everyone would trade in the same way. The point is simply to have access to more information and to derive some logic (fuzzy or solid) that gives you the edge over someone else.

-John-


If you look at what many CEP engines offer to day, most do not offer fuzzy logic out of the box. Look at Tim Bass' blog about CEP and you'll see that the majority of them do not handle fuzzy logic. his blog is here.
http://www.thecepblog.com/

peter

  Message #310488 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Substantiate: "irresponsible," and "what led to the disaster

Posted by: peter lin on June 30, 2009 in response to Message #310438
It's irresponsible to trade on unsubstantiated rumor. Is it done today? Sure it's done, but it's wrong. That type of behavior is a symptom of rampant greed.

I see a balanced view of traders, and trading. Reckless traders exist - agreed. Trading on unsubstantiated rumors is reckless - agreed. Madoff was a crook - agreed.

On the other hand, the vast majority of traders I know are thoughtful, intelligent, and ethical.

Both types of traders - reckless and thoughtful - use news and sentiment to inform their decisions. Sometimes, a thoughtful trader will be informed of sentiment, recognize it as folly, and do nothing. That's a constructive use of news, and it's why Bloomberg, CNN, and other news sources sit on trader desks all over the world. It's why Reuters and Dow Jones transmit their news in machine-readable formats. Twitter's is just another input, with different characteristics. Should we take CNN and Bloomberg off trading floors too?

- Mark


I have met good traders and I agree they aren't all bad. Having said that, there are a lot of shady people in the industry. if you know the boston financial sector, you'll know what happened at the top 5 firms between 2003 and 2007.

Since I worked on pre and post trade compliance, I saw a lot of questionable behavior, which trade side people never see or hear about.

peter

  Message #310489 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Comments on Twitter, from StreamBase

Posted by: John Davies on June 30, 2009 in response to Message #310427
I would argue it's this kind of irresponsible activities that led to the disaster we see today. Great to see technology help people do things of questional ethical and moral value. Even better that vendors are encouraging the same type of questionable risk taking.

If the finance industry insists on such questionable and dubious practices, then I'm sure the government will gladly regulate them. Just because you can, doesn't mean you should.
my bias 2 cents.
peter

Peter,
Trading and making a profit is a fact of human existence, you need to make a profit to survive over the lean times. If a broker or trader didn't make any money then you wouldn't get interest on your pension, banks wouldn't have money to lend so that you can buy houses and cars and industries wouldn't have financing so invest in R&D to develop cool gadgets like the iPhone. That's just the way it is, it's not all ethical I agree and I hope we see more clamp-downs on the less ethical but like it or not Twitter is useful to traders and I can't see that comparing to the likes of Madoff.

-John-

  Message #310490 Post reply Post reply Post reply Go to top Go to top Go to top

Re: News agencies already provide this kind of service

Posted by: John Davies on June 30, 2009 in response to Message #310487
Look at Tim Bass' blog about CEP and you'll see that the majority of them do not handle fuzzy logic. his blog is here.
http://www.thecepblog.com/

peter


Thanks I'll take a look, off for some lunch :-)

-John-

  Message #310492 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Comments on Twitter, from StreamBase

Posted by: peter lin on June 30, 2009 in response to Message #310489
I would argue it's this kind of irresponsible activities that led to the disaster we see today. Great to see technology help people do things of questional ethical and moral value. Even better that vendors are encouraging the same type of questionable risk taking.

If the finance industry insists on such questionable and dubious practices, then I'm sure the government will gladly regulate them. Just because you can, doesn't mean you should.
my bias 2 cents.
peter

Peter,
Trading and making a profit is a fact of human existence, you need to make a profit to survive over the lean times. If a broker or trader didn't make any money then you wouldn't get interest on your pension, banks wouldn't have money to lend so that you can buy houses and cars and industries wouldn't have financing so invest in R&D to develop cool gadgets like the iPhone. That's just the way it is, it's not all ethical I agree and I hope we see more clamp-downs on the less ethical but like it or not Twitter is useful to traders and I can't see that comparing to the likes of Madoff.


-John-


I agree trading is part of life. I'm fine with ethical trading. I agree twitter data "can" be useful. The real question for me is "do these tools really deliver value?" to me, it's mostly hype. I've been studying AI for 8 years now. Fuzzy logic and machine learning are very difficult to implement. I have serious doubts the current CEP market leaders can handle fuzzy logic and machine learning. Most of them are stream processing engines and that's it. They aren't designed for AI.

peter

  Message #310494 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: Benoit Xhenseval on June 30, 2009 in response to Message #310483
During some devnest discussion a "Twitter Manifesto" was discussed in order to clarify some of the processes around Twitter. See http://www.nuxnix.com/images/stories/devnest/mz-twitter-devnest-manifesto-230609-final.pdf

For instance, what is the difference between a "selected/reduced" feed and the full feed? I don't know, none of the devnest attendees knew for sure what it meant.

You are also limited to the number of people you can follow (unless you are a celeb/cnn/etc)

Twitter, rightfully, sees the raw data as their crown jewel but the whole process and ecosystem is still very opaque.

Do you get enough information from the standard feed? StreamBase seems to think so, but it is more like a feeling rather than a metrics or set of statistics to prove it.

In any case, having some information is better than no information, I agree with your examples and also one could use the iphone 3S example eg factoring in the number of iPhone-related tweets around the launch to realise that they were selling like hot cakes... 1 million units over the weekend as it transpired later on... Useful trading information I'd say.

Benoit

  Message #310495 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Comments on Twitter, from StreamBase

Posted by: John Davies on June 30, 2009 in response to Message #310492
I agree trading is part of life. I'm fine with ethical trading. I agree twitter data "can" be useful. The real question for me is "do these tools really deliver value?" to me, it's mostly hype. I've been studying AI for 8 years now. Fuzzy logic and machine learning are very difficult to implement. I have serious doubts the current CEP market leaders can handle fuzzy logic and machine learning. Most of them are stream processing engines and that's it. They aren't designed for AI.

peter


Do they deliver value? Well you'll have to talk to the StreamBase guys about that. I'd guess like any small to medium sized company they're more into providing what customers what than hype, the hype tends to come from the larger companies that don't actually have product or the agility to move their product into the right market so they create hype around what they do do like "SOA" and other related crap.

It's interesting to see companies like Microsoft, IBM and RedHat getting into CEP with what's little more than a rules engine. Of course their marketing people will tell you it's the latest and greatest but you can be sure it wasn't designed to do what it's doing today or as in the case of the first two, only runs on their propitiatory OS or kit.

CEP is not necessarily AI, it's not a rules engine either, they are all different things. The origin of CEP tends to be, in my own opinion, from algorithmic trading and unless it's from that domain I really don't think you can fit a rules engine into the CEP category. AI on the other hand has a fairly long relationship with trading so I guess there's an element of overlap. Given that a good CEP product is really just a application with programming tools then AI could/would be part of that.

A lot of the work I do for clients involves very high performance (and volume) processing so I come across the need for rules engines and CEP pretty frequently. AI is a big part of that but what I look for in an engine (CEP or rules) is a flexible DSL or language that is easy to recruit for and obviously a scalable architecture.

-John-

  Message #310498 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Comments on Twitter, from StreamBase

Posted by: peter lin on June 30, 2009 in response to Message #310495
I agree trading is part of life. I'm fine with ethical trading. I agree twitter data "can" be useful. The real question for me is "do these tools really deliver value?" to me, it's mostly hype. I've been studying AI for 8 years now. Fuzzy logic and machine learning are very difficult to implement. I have serious doubts the current CEP market leaders can handle fuzzy logic and machine learning. Most of them are stream processing engines and that's it. They aren't designed for AI.

peter


Do they deliver value? Well you'll have to talk to the StreamBase guys about that. I'd guess like any small to medium sized company they're more into providing what customers what than hype, the hype tends to come from the larger companies that don't actually have product or the agility to move their product into the right market so they create hype around what they do do like "SOA" and other related crap.

It's interesting to see companies like Microsoft, IBM and RedHat getting into CEP with what's little more than a rules engine. Of course their marketing people will tell you it's the latest and greatest but you can be sure it wasn't designed to do what it's doing today or as in the case of the first two, only runs on their propitiatory OS or kit.

CEP is not necessarily AI, it's not a rules engine either, they are all different things. The origin of CEP tends to be, in my own opinion, from algorithmic trading and unless it's from that domain I really don't think you can fit a rules engine into the CEP category. AI on the other hand has a fairly long relationship with trading so I guess there's an element of overlap. Given that a good CEP product is really just a application with programming tools then AI could/would be part of that.

A lot of the work I do for clients involves very high performance (and volume) processing so I come across the need for rules engines and CEP pretty frequently. AI is a big part of that but what I look for in an engine (CEP or rules) is a flexible DSL or language that is easy to recruit for and obviously a scalable architecture.

-John-


I agree a CEP engine is different than a general purpose business rule engine, which is also different than AI. My point is that using twitter feeds with the existing CEP engines is impractical, since they do not support fuzzy logic. I've dabbled with FuzzyJess and studied fuzzy logic. I'm not an expert or intermediate user of fuzzy logic. What I do know is that it's very hard to implement effectively and isn't something an average developer can learn in a few months or even 1-2 years.

it would be trivial for a person to use zombie systems to generate tweet traffic about a company or stock and start a chain reaction. My fear irrational as it may be is that once a trader starts to use twitter data, someone else will exploit that. A lot of by standers may end up getting hurt by inappropriate use of technology.

for the record, I feel twitter-like data may be useful in the future, but right now the noise is far too high.

peter

  Message #310503 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: James Watson on June 30, 2009 in response to Message #310486
I see a lot of problems with this. First and foremost, this kind of service seems ready-made for market manipulation schemes. Secondly, isn't Twitter famously prone to being overwhelmed? And lastly, if this financial news is so important, why would someone expose it in real time instead of taking advantage of it? They aren't getting paid to do so. Are they just stupid? Or are they selfless saps that want to help other people get/stay rich?
On a side note, what about Twitter is special? I don't see much about their model that can't be stolen and improved upon. The only thing that they have is membership and the day that someone provides aggregation for this type of service, that's not going to matter.

James,
With all due respect I think you're missing the point. Trades are essentially gambling on the way a market will go, up, down etc., much the same way someone betting on a horse to win.


And that it exactly what's wrong with the stock market. People don't care about owning companies anymore. It's just a big (legal) online casino.

People who've done good research tend to have the edge on someone who just likes the name of the horse, someone who knows the trainer and the jockey have an even better edge, any more and you're getting into illegal betting or insider trading as it's known in the trading world.

So, if you can improve the source of information then you have an edge over the other traders. Take the example of a Reuters feed announcing a missing plane (not the nicest example I know, sorry), the fact is the plane's missing, nothing else is known, you can speculate on that news and relate it to travel shares. Now you see some noise on Twitter about the airline or the fact that it landed safely on a beach (I was looking for a happy ending), news agencies like Reuters won't announce this until it's conformed but you'll get some noise on Twitter some time before and you can use that to trade your shares before the "official" announcement comes out.


What exactly makes you think I don't understand something such painfully simple concept? Don't you see that this can go the other way?

Consider this:
Six-Year-Old News Story Causes United Airlines Stock to Plummet — UPDATE Google Placed Wrong Date on Story

Did the people that dumped their United Airlines stock at nearly $3 (75% loss) benefit from this information? I guess your argument is that the people who got out at the top gained from knowing the false information first. When I see a problem, I don't think, "hey how can I make this problem worse in order to profit" but that's just me.

Why should we believe that this kind of thing won't happen with Twitter? How hard would it be to hire a bunch of people to create Twitter accounts and all start 'twittering' about a given stock? Are you not familiar with pump and dump schemes and other similar techniques? We've all seen now that the smart money is pretty stupid. I suggest you read up on the history of the stock market and why news on stocks is so highly regulated.

What inspired me to "blog" about this on TSS was the news of MJ's death, I first heard about it on Twitter some time before the BBC (my reliable choice of news feed) or CNN any other "reliable" feed broke the news. There was also interesting (sad again) speculation about Jeff Goldblum having died in New Zealand on Twitter, now whether that happened of not (and fortunately it seems like it didn't) people might have started trading the shares of the investment company sponsoring his latest film, if I was a trader (and I'm not), I'd want the opportunity to make money on such things. Streambase's ability to use Twitter as a source of "fuzzy" information would increase my edge on the rest of the market.


You could also lose money on such things and the reality is that this asinine bullshit has a real effect on companies and the economy. If I had my druthers, taxes on short term gains would be set to excruciatingly costly levels.

What makes Twitter so special, nothing, you could set up the same thing, problem is that Twitter's the one people use and you're unlikely to get anywhere with your attempt. If you think you can improve on it then good luck, sell me on the idea and I'll invest some money.

-John-


Why do people repeat back to me what I just said to them as if they are telling me something I don't know?

  Message #310502 Post reply Post reply Post reply Go to top Go to top Go to top

On a more humurous note, even the dead can tweet

Posted by: peter lin on June 30, 2009 in response to Message #310401
As jeff goldblum shows us, even the dead and twitter
http://news.yahoo.com/s/eonline/20090630/en_top_eo/131952

  Message #310506 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: James Watson on June 30, 2009 in response to Message #310486
What makes Twitter so special, nothing, you could set up the same thing, problem is that Twitter's the one people use and you're unlikely to get anywhere with your attempt.


Exactly. Just like AIM or AOL in general. Remember when AOL was the a sure bet?

If you think you can improve on it then good luck, sell me on the idea and I'll invest some money.

-John-


I don't really see myself being an outright idea thief. I guess that's a personal flaw. I just don't see how Twitter is ever going to make a profit. Based on your logic Hulu could not manage to come into prominence in the shadow of YouTube. Firefox could never steal IE's market share. The reality is that it's no harder to get 'tweets' off of any other similar service and I doubt it will be long before no one will care about who distributes their 'tweets' any more than we care about how our email gets to it's destination.

  Message #310518 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: John Davies on June 30, 2009 in response to Message #310506
What makes Twitter so special, nothing, you could set up the same thing, problem is that Twitter's the one people use and you're unlikely to get anywhere with your attempt.
Exactly. Just like AIM or AOL in general. Remember when AOL was the a sure bet?
If you think you can improve on it then good luck, sell me on the idea and I'll invest some money.

-John-


I don't really see myself being an outright idea thief. I guess that's a personal flaw. I just don't see how Twitter is ever going to make a profit. Based on your logic Hulu could not manage to come into prominence in the shadow of YouTube. Firefox could never steal IE's market share. The reality is that it's no harder to get 'tweets' off of any other similar service and I doubt it will be long before no one will care about who distributes their 'tweets' any more than we care about how our email gets to it's destination.


I won't even bother to answer your other post, obviously you have a grudge to bear so I'll leave you with your troubles.

I never liked AOL, I don't really care about whether Twitter make money form their business or not as I don't own any equity but one thing is for sure, they have perceived value and that will probably make them a fortune if they sell the company. I wish them luck as I do any entrepreneurs.

If you can provide value on top of an already commoditised market place then people will buy your product, one day someone may do that with Tweets, we shall see.

-John-

  Message #310520 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: James Watson on June 30, 2009 in response to Message #310518
I won't even bother to answer your other post, obviously you have a grudge to bear so I'll leave you with your troubles.


And ad homimen argument. Instead of responding to a challenge, just attempt to make the challenge appear illegitimate. Is that really the best you can come up with?

  Message #310522 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: James Watson on June 30, 2009 in response to Message #310518
I never liked AOL, I don't really care about whether Twitter make money form their business or not as I don't own any equity but one thing is for sure, they have perceived value and that will probably make them a fortune if they sell the company.


And isn't that what it's all these days? Value is not important, it's perceived value. For example, a person can be a complete blowhard and not know what they are talking about but if he or she sounds like they know what they are talking about, that person is considered an expert. pets.com had a lot of 'perceived value' once upon a time. A lot of people are struggling now because the 'perceived value' of their homes has changed drastically over a short period of time. Sure, if you can pass a cubic zirconium off as a diamond, hooray for you. That doesn't mean I have to pretend it's a diamond, though. I just call em like I see em.

  Message #310541 Post reply Post reply Post reply Go to top Go to top Go to top

GOOD

Posted by: jock guck on July 01, 2009 in response to Message #310348
tiffany jewelry is the best website of Tiffany & Co Jewelry Stores.Every year,the sales volume of Tiffany Jewelry online surpasses 10000.
Tiffany Bracelets for the best prices, guaranteed! Tiffany provides the best Tiffany & Co Sterling Silver Bracelets to our customers at the
tiffany co jewelry
Tiffany Rings
cheap tiffany
Tiffany & Co Jewelry Outlet Online Store Sells Various Kinds of Cheap Tiffany's Silver Jewelry, Including
Tiffany Earrings
Tiffany Necklaces
links of london

  Message #310559 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: John Davies on July 01, 2009 in response to Message #310520
I won't even bother to answer your other post, obviously you have a grudge to bear so I'll leave you with your troubles.


And ad homimen argument. Instead of responding to a challenge, just attempt to make the challenge appear illegitimate. Is that really the best you can come up with?

My intention was to talk about clever technology not your rants about the ethics of trading.

-John-

  Message #310562 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: John Davies on July 01, 2009 in response to Message #310522
A lot of people are struggling now because the 'perceived value' of their homes has changed drastically over a short period of time.

It's not the perceived value of their homes it's the perceived ability of the tenant to pay back the loans. If someone with little education, a bad job and a history of debts (i.e. sub-prime) is lent money it doesn't take a rocket scientist to understand they might have a problem paying back the loan. The value given to the house was what the tenant was prepared to pay, since they had little hope of ever paying back the money they were happy to pay any price as it was effectively free money. Only when people began to unravel the sub-prime business did the true value of the properties emerge, that price dropped further when fewer people were prepared to buy.

Who's fault was it? Unscrupulous mortgage companies, lack of regulation and erroneous risk ratings. If all this had been in place though the US might have had an equally large problem housing and taking care of the millions of poor people. Give the poor a relatively cheap house and endless credit cards that they never pay back and you don't have to worry about paying their medical bills, feeding or housing them. They continue to buy cars and gas on credit and the car industry is supported by the same sub-prime money. Look at it this way and you can start to see why no one bothered to blow the whistle earlier.

Back to Streambase and Twitter, this has nothing to do with the above, this will be my last post on this subject, this is a technology forum and I'd rather talk about the technology behind trading engines, FIX engines and algorithmic trading.

-John- (sitting in the very warm sun drinking a nice green tea in London).

  Message #310569 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: peter lin on July 01, 2009 in response to Message #310562
Back to Streambase and Twitter, this has nothing to do with the above, this will be my last post on this subject, this is a technology forum and I'd rather talk about the technology behind trading engines, FIX engines and algorithmic trading.

-John- (sitting in the very warm sun drinking a nice green tea in London).


If you want to discuss the technological side of CEP, you can look at the CEP forum http://forum.complexevents.com/viewforum.php?f=13. In the past, I worked on a natural-language buisness rule language for pre/post trade compliance. In one of your responses, you mentioned DSL. If you want to continue the discussion, email me at woolfel AT gmail DOT com. I'd be interested to see what kind of DSL you're thinking and compare it to what I've done before.

peter

  Message #310576 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: James Watson on July 01, 2009 in response to Message #310562
A lot of people are struggling now because the 'perceived value' of their homes has changed drastically over a short period of time.

It's not the perceived value of their homes it's the perceived ability of the tenant to pay back the loans. If someone with little education, a bad job and a history of debts (i.e. sub-prime) is lent money it doesn't take a rocket scientist to understand they might have a problem paying back the loan. The value given to the house was what the tenant was prepared to pay, since they had little hope of ever paying back the money they were happy to pay any price as it was effectively free money.


That demonstrates an incomplete understanding of the issues. No one said to themselves "I want to buy a house and have it go into foreclosure in 5 years." You can't take off with a house in-tow. The idea that large numbers of people got mortgages that they didn't intend to pay off is ludicrous.

Having lived in an area where home prices inflated rapidly (and then moving to one where they were depressed before purchasing a home) I can tell you that the perceived value was at the root of the problem. It was a bubble just like all the preceding bubbles. People bought homes they could not afford because they believed they would be able to sell or refinance before their adjustable rates kicked in. The expectation was that the value would rise giving them equity which would allow them to get low fixed rate loans or flip the home and use the equity as a down-payment towards a different home.

This worked for a number of years as values doubled twice over the span of 5 years in that area and continued to rise. When the prices started to drop, however, people could no longer refinance or sell because they were underwater in their loans. The no-interest (or low fixed rate interest) period of their loans expired and their payments increased. Part of the ignition was that interest rates started rising and this resulted in many peoples monthly payments increasing drastically. In some cases they more than doubled.

I actually know someone who bought a house that he could not afford using an interest-only loan and this was the exact reasoning. It wasn't his idea, this is what the 'experts' were saying was a smart move because "real estate doesn't lose value". Even I, who had never even purchased a home at the time, could see that the prices were unsustainable but the 'perceived value' (a friend once called it pixie-dust money) trumped reason. People told me I was crazy when I moved to one of the most depressed (comparatively) real-estate markets to buy a home. It's insane to think that a town-house a stone's throw from a subway station/bus-stop/congested interstate is worth a nearly a million dollars but people figured that it would be worth 2 million before long.

And on a side note, this was not a US only problem. Real-estate prices were inflated in many parts of the world.

http://www.economist.com/markets/rankings/displaystory.cfm?story_id=13337869&CFID=68027530&CFTOKEN=28493414

  Message #310577 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: James Watson on July 01, 2009 in response to Message #310559
I won't even bother to answer your other post, obviously you have a grudge to bear so I'll leave you with your troubles.


And ad homimen argument. Instead of responding to a challenge, just attempt to make the challenge appear illegitimate. Is that really the best you can come up with?

My intention was to talk about clever technology not your rants about the ethics of trading.

-John-


I was talking about the ethics of the clever technology you are proposing and how it appears to attempt to circumvent the regulations you now claim to be keen on.

  Message #310593 Post reply Post reply Post reply Go to top Go to top Go to top

How to use rel=nofollow attribute to discourage SEO spammers

Posted by: Casual Visitor on July 01, 2009 in response to Message #310541
http://www.byteblocks.com/post/2009/06/30/Use-rel3dnofollow-to-discourage-SEO-spammers.aspx

  Message #310598 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: John Davies on July 01, 2009 in response to Message #310569
If you want to discuss the technological side of CEP, you can look at the CEP forum http://forum.complexevents.com/viewforum.php?f=13. In the past, I worked on a natural-language buisness rule language for pre/post trade compliance. In one of your responses, you mentioned DSL. If you want to continue the discussion, email me at woolfel AT gmail DOT com. I'd be interested to see what kind of DSL you're thinking and compare it to what I've done before.

peter

Thanks, I've taken a look around the forum and I'll drop you a line some time. My email is John dot Davies at OnixS dot biz.

-John-

  Message #310606 Post reply Post reply Post reply Go to top Go to top Go to top

Of course this is happening - technical detail?

Posted by: Jim Walsh on July 02, 2009 in response to Message #310348
Of course Twitter as a information source is a good idea. No different than having CNN on the trader's desk. Let's get off the "traders are evil" rants and back to technology....

What kind of connectivity does Twitter allow and how do you link it to StreamBase? What kind of data rates can you push through? John, do you use Twitter and how?

- Jim

  Message #310617 Post reply Post reply Post reply Go to top Go to top Go to top

Re: News agencies already provide this kind of service

Posted by: Tim Bass on July 02, 2009 in response to Message #310484
The risk with using tweets is the data is not reliable, so you need some pretty sophisticated fuzzy logic functions to make the news useful.
my bias 2 cents
peter

Peter,
The fuzzy logic is exactly what CEP companies specialise in, if everyone used a feed of 100% reliable data then everyone would trade in the same way. The point is simply to have access to more information and to derive some logic (fuzzy or solid) that gives you the edge over someone else.

-John-


Hi John,

I was surprised to read such misinformation in your blog.

First of all, you say "The fuzzy logic is exactly what CEP companies specialise in."

How can you publish this? There are no companies in the CEP space that specialize in fuzzy logic.

To process text required very sophisticated techniques, not query-based rule-style processing. If it was easy to build systems that could process text from unreliable sources, then folks would have done this years ago.

I am really surprised to read such misinformation, quite frankly. More surprising is that this is a popular blog and so many people have responded, demonstrating what a wacky world we live in.

Yours, Tim

  Message #310618 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Comments on Twitter, from StreamBase

Posted by: Tim Bass on July 02, 2009 in response to Message #310492
I would argue it's this kind of irresponsible activities that led to the disaster we see today. Great to see technology help people do things of questional ethical and moral value. Even better that vendors are encouraging the same type of questionable risk taking.

If the finance industry insists on such questionable and dubious practices, then I'm sure the government will gladly regulate them. Just because you can, doesn't mean you should.
my bias 2 cents.
peter

Peter,
Trading and making a profit is a fact of human existence, you need to make a profit to survive over the lean times. If a broker or trader didn't make any money then you wouldn't get interest on your pension, banks wouldn't have money to lend so that you can buy houses and cars and industries wouldn't have financing so invest in R&D to develop cool gadgets like the iPhone. That's just the way it is, it's not all ethical I agree and I hope we see more clamp-downs on the less ethical but like it or not Twitter is useful to traders and I can't see that comparing to the likes of Madoff.


-John-


I agree trading is part of life. I'm fine with ethical trading. I agree twitter data "can" be useful. The real question for me is "do these tools really deliver value?" to me, it's mostly hype. I've been studying AI for 8 years now. Fuzzy logic and machine learning are very difficult to implement. I have serious doubts the current CEP market leaders can handle fuzzy logic and machine learning. Most of them are stream processing engines and that's it. They aren't designed for AI.

peter


I agree with Peter. Folks get both emotional and excited about technology, blinding them to actually understanding what they are talking about. That's OK, but there is also humor in it. Seriously,

It takes very sophisticated processing to take textual data from myriad untrusted sources and determine anything of value. AI folks have struggled with this for decades, as Peter points out. Governments have struggled with this for decades in their intelligence processing systems.

So, being people of logic and human intelligence, how can query-based stream processors that has no sophisticated logic (AI, fuzzy, Baysian, neural or otherwise) compared to decades of technology development and implementation by some of the world's best and brightest engineers and scientist accomplish what no system has reliably accomplished before? (Include Star Trek Theme Music)....

The short answer: It can't.

Anyone who believes that simple query-oriented stream processing engines can process twitter tweeks, which can (must) be modeled as an unreliable, untrusted information channel with a very low signal to noise ratio, does not understand the math and science behind signal processing and related science.

IF twitter was from trusted sources and IF there was an agree upon semantic structure, and IF the channel was reliable and IF ... IF ... IF ....

As I said in my comment to one of Streambase's evangelists:

“Announcement: Streambase has processing intelligence that no other company in the world has, no matter how great or big. We have the “Holy Grail Stream Processing Engine. You can buy it, plug it in, process Twitter rumors, and never have to work another day in your life!

From this post and comments:

http://www.thecepblog.com/2009/06/29/gimmick-marketing-why/

Cheers, Tim

  Message #310622 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Of course this is happening - technical detail?

Posted by: John Davies on July 02, 2009 in response to Message #310606
Of course Twitter as a information source is a good idea. No different than having CNN on the trader's desk. Let's get off the "traders are evil" rants and back to technology....

What kind of connectivity does Twitter allow and how do you link it to StreamBase? What kind of data rates can you push through? John, do you use Twitter and how?

- Jim

Personally I use Twitter yes, I don't trade though and I don't work for or with Streambase so you'll have to wait for a Streambase authority to answer detailed questions. I work in the trading environment and all of my customers are traders or exchanges so I'm very close to the "coal front" so to speak. I just thought it was an interesting development and thought Streambase were being rather clever and forward thinking with their announcement.

-John-

  Message #310624 Post reply Post reply Post reply Go to top Go to top Go to top

Re: News agencies already provide this kind of service

Posted by: John Davies on July 02, 2009 in response to Message #310617
Hi John,

I was surprised to read such misinformation in your blog.

First of all, you say "The fuzzy logic is exactly what CEP companies specialise in."

How can you publish this? There are no companies in the CEP space that specialize in fuzzy logic.

To process text required very sophisticated techniques, not query-based rule-style processing. If it was easy to build systems that could process text from unreliable sources, then folks would have done this years ago.

I am really surprised to read such misinformation, quite frankly. More surprising is that this is a popular blog and so many people have responded, demonstrating what a wacky world we live in.

Yours, Tim

Hi Tim,
Sorry you seem to think this is misinformation, rules used by traders include "fuzzy" logic, my use of the term fuzzy is anything that isn't exact. Having said that, logic can be defined so it's exact, therefore it can't be fuzzy, a contradiction I know. What I meant was that it is quite feasible for a trader to define a rule where...

ORCL price greater than 1% above 15 minute moving average and NASDAQ index steady and JAVA (that's Sun) below 1% of opening price and Twitter count of Oracle from TwitterNewsFeeds double usual volume then buy ORCL

Something like that, my use of the term fuzzy is the "Twitter count of Oracle from TwitterNewsFeeds double usual volume", it's not an exact science at this point.

There's no exact science in trading, if there was there'd be no point in trading.

You might feel this is a waste of TSS space but Streambase is written in Java, many of our clients use Java (as well as our C and .NET engines), I'm a technologist and I though it was all pretty damn cool because I was speaking to clients who regularly trade billions of dollars (and Stirling, Yen, Euro, Oil, Gold etc.) and they're interested in this too. My interest is in the technology behind it.

-John-

  Message #310625 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Of course this is happening - technical detail?

Posted by: James Watson on July 02, 2009 in response to Message #310606
Of course Twitter as a information source is a good idea. No different than having CNN on the trader's desk. Let's get off the "traders are evil" rants and back to technology...


It's not the same as CNN. That's my point and I never said anyone was evil. I implied that some people that are traders are stupid and I stand by that. Maybe you and John could quit with the rhetoric and try to understand.

  Message #310631 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Algorithmic trading with Twitter

Posted by: Louie Lovas on July 02, 2009 in response to Message #310348
Interesting commentary, the AI stuff (IMHO) is getting a bit too esoteric for my taste. CEP is good at bringing multiple streams of data together to correlate patterns - in this case news and trading. This is a common design pattern we've (at Apama) used for quite come time with Dow Jones New Service. The 'challenge' of doing the same with Twitter is latency and validity.

DJ/Reuters are sanctioned news sources focused on low-latency and pride themselves on accuracy in reporting. Neither of these can be applied to information on Twitter.

Here's a couple of others making comments on Twitter as a news source:

http://www.alacrablog.com/alacrablog/2009/06/twitter-is-the-herd-or-trading-with-twitter.html

http://www.marketwatch.com/story/as-news-source-twitter-has-seven-deadly-flaws

  Message #310639 Post reply Post reply Post reply Go to top Go to top Go to top

Re: Of course this is happening - technical detail?

Posted by: peter lin on July 02, 2009 in response to Message #310625
Of course Twitter as a information source is a good idea. No different than having CNN on the trader's desk. Let's get off the "traders are evil" rants and back to technology...


It's not the same as CNN. That's my point and I never said anyone was evil. I implied that some people that are traders are stupid and I stand by that. Maybe you and John could quit with the rhetoric and try to understand.


Actually, it was me who suggested "some" traders are evil. I stand by that. Not all traders are evil.

Back to technology. What most CEP engines do today I classify under decision automation. It uses simplistic rules, with a fixed algorithm for deciding when to trade a security, or how to execute the trade to optimize the cost. One obvious example is the volume weighted average price, which is only good for executing trades once a human being decides to purchase X dollars of Y stock.

What business rules don't handle well is deciphering what triggered the change. To do that, the system has to have a knowledge base and utilize fuzzy logic to make a "good guess". What some vendors are calling knowledge base today is just a business dictionary. Within AI there's the concept of strong negation, weak negation, unknown, and unknowable. For example, a system that tries to model or predict stocks can't say with 100% certainty why a stock is moving up or down.

That means the system has to support the notion of degrees of truth. The rules a developer or business person would write has to be in those terms. It gets even more complex when you take into account the system has to change and adapt on the fly to changing input. If a developer hard codes a function, that probably won't work, since the environment is highly dynamic.

In my bias opinion, the current crop of event processing engines are not powerful enough to handle these types of use cases. I hope one day they will.

peter

New content on TheServerSide.comNew content on TheServerSide.comNew content on TheServerSide.com

Dependency Injection in Java EE 6 - Part 1

Reza Rahman explores the features of the proposed JSR 299, Contexts and Dependency Injection for Java EE (CDI). When approved, it promises to be a key feature of Java EE 6. (November 2, Article)

SAML: It's Not just for Web services

SAML is an XML-based standard for exchanging authentication and authorization data between security domains. The single most important problem that SAML was created to solve is the Web browser Single Sign-On problem. Many organizations are debating whether to stay with version 1.1 or move to 2.0. This article makes observations about both options. (September 28, Article)

Programming is Also Teaching Your Team

Joe Ottinger takes a look at how people learn, and applies it to the practice of programming. He notes that understanding how people learn is an essential part of working in a programming team. (September 22, Article)

Can Java EE Deliver The Asynchronous Web?

Stephen Maryka gave us an article about the Asynchronous Web and posed a number of questions that get examined like an approach to delivering Asynchronous Web capabilities through extensions to existing Java EE technologies. (July 14, Article)

JSF Flex

JavaServer Faces Flex goal is to provide users capability in creating standard Flex components, part of flexSDK which is open sourced through MPL license, as normal JSF components. This article by Ji Hoon Kim will provide an overview of creating a simple multilingual JSF page consisting of JSF Flex tags. (June 29, Article)

The Rules of SOA - A Road to a Successful SOA Implementation

In this session Jeff explores the key characteristics of successful SOA projects. He covers some of the patterns, and anti-patterns, tool sets, and strategies that he himself learned the hard way. Last, he provides a strategy and blueprint for achieving a high likelihood of success in your SOA project. (June 23, Tech Talk)

Ari Zilka Talks About Terracotta 3.1

Ari Zilka, CTO of Terracotta, Inc., talks about the new features in Terracotta 3.1, announced during JavaOne and available now. (June 15, Tech Talk)

Enterprise Application Integration, and Spring

In this Tech Talk, Josh Long explores an integration challenge using Spring Integration and walks through the implementation, employing and expanding on the basic patterns of Enterprise Application Integration to tie together components into a function integration solution, and then demonstrates how Spring Integration helps address the integration requirements. (June 15, Tech Talk)

Google Web Toolkit: An Introduction

In this Tech Talk, David Geary teaches you: The basics of Google Web Toolkit; How to implement Ajax-enabled applications in Java; Internationalization; Hooking into the browser history mechanism; Remote procedure calls. (June 4, Tech Talk)

Just Enough Early Architecture to Guide Development

Jon Kern discusses the best architecture/technical solutions and ensure that they are repeated by all developers. By tackling the architecture up-front in a serial manner, subsequent parallel development will be much more manageable and predictable. (May 28, Tech Talk)

Productive Programmer: On the Lam from the Furniture Police

This keynote describes the frustrations of modern knowledge workers in their quest to actually get some work done, and solutions for how to guard yourself against all those distractions. Neal Ford talks about environments, coding, acceleration, automation, and avoiding repetition as ways to defeat the misguided attempts to sap your ability to produce good work. (May 26, Tech Talk)

Auto-Scaling Your Existing Web Application

Gil demonstrates how new, aggressive uses of already abundant compute capacity by common applications offer competitive value for application designers. (May 21, Tech Talk)

Automating Hibernate Mapping and Queries For Java Web Development

Chris Keene introduces WaveMaker as a new way to automate the ability to generate Hibernate classes in order to more quickly bring OR mapping into an application. (May 19, Article)

Auto-Scaling Your Existing Web Application

In this session Nati Shalom demonstrates how to take a standard Java EE web application and scale it out or down dynamically without changes to the application code. Seeing as most web applications are over-provisioned to meet infrequent peak loads, this is a dramatic change because it enables growing your application as needed, when needed, without paying for unutilized resources. (May 19, Tech Talk)

Free Book: Jakarta-Struts Live

Download the entire book of Jakarta-Struts Live and learn about Struts MVC, Tiles, the Validator, DynaActionForms, plug-ins, internationalization, and more.
(Book PDF Download)

Application Server Matrix

The Application Server Matrix is a detailed listing of J2EE vendors and their application server products, with information on latest version numbers, J2EE spec support and licensing, pricing, platform support, and links to product downloads and reviews.
(Application Server Comparison Matrix)

News | Blogs | Discussions | Tech talks | Patterns | Reviews | White Papers | Downloads | Articles | Media kit | About
Java Solutions
All Content Copyright ©2007 TheServerSide Privacy Policy
Site Map