Support Forums: Message List - Language Identification System: How to recognize other languages than English http://www.theserverside.com Most recent forum messages en Jive Forums Silver 5.5.30 (www.jivesoftware.com) Thu, 23 May 2013 07:22:26 -0400 Language Identification System: How to recognize other languages than English http://www.theserverside.com/discussions/thread.tss?thread_id=62570 Analyzing short texts will always be a major headache as they contain so little information. Regarding the Twitter case, it may be a good idea to add multiple posts from the same user together. This way you get more words. And one user probably uses...]]> Fri, 01 Mar 2013 08:01:42 -0500 Fri, 01 Mar 2013 08:01:42 -0500 Fri, 01 Mar 2013 08:01:42 -0500 Mar 1, 2013 Nick Snels 0 Re: Language Identification System: How to recognize other langua http://www.theserverside.com/discussions/thread.tss?thread_id=62570 "Are you using Levenshtein's distance?"

 

I do. Sometimes, depending on ngram analysis :)

]]> Wed, 29 Jun 2011 12:48:20 -0400 Wed, 29 Jun 2011 12:48:20 -0400 Wed, 29 Jun 2011 12:48:20 -0400 Jun 29, 2011 Istvan Soos 0 Re: Language Identification System: How to recognize other langua http://www.theserverside.com/discussions/thread.tss?thread_id=62570 Hi Istvan,

...]]> Wed, 29 Jun 2011 11:23:50 -0400 Wed, 29 Jun 2011 11:23:50 -0400 Wed, 29 Jun 2011 11:23:50 -0400 Jun 29, 2011 israel olalla 1 Language Identification System: How to recognize other languages than English http://www.theserverside.com/discussions/thread.tss?thread_id=62570 Snowball might be good for some languages, stopwords for a different (slightly overlapping) set of languages, but in general languages, you need to find other solution. For example variable-sized ngram with the combination of a text-distance algorithm...]]> Wed, 29 Jun 2011 05:49:52 -0400 Wed, 29 Jun 2011 05:49:52 -0400 Wed, 29 Jun 2011 05:49:52 -0400 Jun 29, 2011 Istvan Soos 2 Language Identification System: How to recognize other languages than English http://www.theserverside.com/discussions/thread.tss?thread_id=62570 Language identification systems usually fail when we are analyzing short sentences, from the solutions based in ngram to solutions based in dictionaries, usually fail when analyzing short Sentences, but we've arranged a new technique based in stemming...]]> Tue, 28 Jun 2011 07:56:03 -0400 Tue, 28 Jun 2011 07:56:03 -0400 Tue, 28 Jun 2011 07:56:03 -0400 Jun 28, 2011 israel olalla 4