Comments on: How to convert html entities to “real” unicode in Python http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/ A blog about SecondLife, webservices, and whatever else seems to fit. Fri, 16 Oct 2009 01:42:45 +0000 http://wordpress.com/ hourly 1 By: LlucPot http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-866 LlucPot Fri, 02 Oct 2009 11:03:29 +0000 http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-866 from BeautifulSoup import BeautifulStoneSoup def HTMLtoUni(entity): uni=str() if type(entity)!= type(str()): ## entity is not a string return if not re.match(u'&[#a-z0-9]+;',entity): ##entity is not an HTML entity return if re.match(u'&#x[a-f0-9]+;',entity): ## convert hex HTML entity to HTML decimal entity entity=str('&#%i;'%int('0%s'%entity[2:-1],16)) return unicode(BeautifulStoneSoup(entity,convertEntities=BeautifulStoneSoup.HTML_ENTITIES )) from BeautifulSoup import BeautifulStoneSoup

def HTMLtoUni(entity):
uni=str()
if type(entity)!= type(str()):
## entity is not a string
return
if not re.match(u’&[#a-z0-9]+;’,entity):
##entity is not an HTML entity
return
if re.match(u’&#x[a-f0-9]+;’,entity):
## convert hex HTML entity to HTML decimal entity
entity=str(‘&#%i;’%int(‘0%s’%entity[2:-1],16))
return unicode(BeautifulStoneSoup(entity,convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))

]]> By: LlucPot http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-865 LlucPot Fri, 02 Oct 2009 10:51:27 +0000 http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-865 Solved: hex entity to decimal entity: ## IF you have encodedStringHex ## DO(in Python): encodedStringDecimal=str('&#%i;'%int('0%s'%encodedStringHex[2:-1],16)) ##THEN the solution posted above will work Solved:
hex entity to decimal entity:

## IF you have encodedStringHex
## DO(in Python):
encodedStringDecimal=str(‘&#%i;’%int(‘0%s’%encodedStringHex[2:-1],16))
##THEN the solution posted above will work

]]> By: LlucPot http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-864 LlucPot Fri, 02 Oct 2009 10:27:10 +0000 http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-864 Tough it's a helpul piece of advice, I'm in a trickier situation. I have to convert HTML entities IN HEX to uni. So BeautifulSoup complains that I give to it an 'invalid literal for int() with base 10'. The entities I face are the type: {&#x~hex number here~;} (i.e.: '&#xf2' for '&242'/'ò'). Does anyone knows soemthing about it? Tough it’s a helpul piece of advice, I’m in a trickier situation.

I have to convert HTML entities IN HEX to uni. So BeautifulSoup complains that I give to it an ‘invalid literal for int() with base 10′. The entities I face are the type: {&#x~hex number here~;} (i.e.: ‘&#xf2′ for ‘&242′/’ò’). Does anyone knows soemthing about it?

]]> By: javaJake http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-863 javaJake Mon, 14 Sep 2009 21:46:46 +0000 http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-863 Thanks ciemaar for this blog post! Thanks ciemaar for this blog post!

]]> By: Jonas Byström http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-842 Jonas Byström Tue, 31 Mar 2009 09:46:03 +0000 http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-842 Thanks a million! Yeah, amazing they slipped on this one. Thanks a million! Yeah, amazing they slipped on this one.

]]> By: Ian Young http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-838 Ian Young Sat, 22 Nov 2008 03:26:28 +0000 http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-838 "Beautiful Soup uses a class called UnicodeDammit to detect the encodings of documents you give it and convert them to Unicode, no matter what. If you need to do this for other documents (without using Beautiful Soup to parse them), you can use UnicodeDammit by itself." http://www.crummy.com/software/BeautifulSoup/documentation.html#Beautiful%20Soup%20Gives%20You%20Unicode,%20Dammit I haven't tried using the class by itself, but it sounds like that could be the solution to the not-needing-the-whole-library woes. “Beautiful Soup uses a class called UnicodeDammit to detect the encodings of documents you give it and convert them to Unicode, no matter what. If you need to do this for other documents (without using Beautiful Soup to parse them), you can use UnicodeDammit by itself.”

http://www.crummy.com/software/BeautifulSoup/documentation.html#Beautiful%20Soup%20Gives%20You%20Unicode,%20Dammit

I haven’t tried using the class by itself, but it sounds like that could be the solution to the not-needing-the-whole-library woes.

]]> By: M http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-837 M Tue, 04 Nov 2008 17:35:28 +0000 http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-837 I did what rabio suggested and it works perfectly! Thanks a lot! I did what rabio suggested and it works perfectly! Thanks a lot!

]]> By: Tørbjorn http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-836 Tørbjorn Tue, 14 Oct 2008 08:15:29 +0000 http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-836 I just wanna join in on the praise of the article, it really helped me out too. BeautifulSoup is a hell of library, and it has saved my more times than I want to remember. On the other hand, why is coding and deconding HTML / XML entities so "hard" in python ? I just wanna join in on the praise of the article, it really helped me out too.
BeautifulSoup is a hell of library, and it has saved my more times than I want to remember.

On the other hand, why is coding and deconding HTML / XML entities so “hard” in python ?

]]> By: akahn http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-835 akahn Mon, 06 Oct 2008 21:57:12 +0000 http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-835 Looks like Beautiful Soup is a solid library, but it seems like overkill for my usage, unfortunately. In my sub-300 line script, I'm trying to decode 160 character strings, 20 at a time, so using this whole library seems wrong... maybe I'll do something like rabio suggests? Looks like Beautiful Soup is a solid library, but it seems like overkill for my usage, unfortunately. In my sub-300 line script, I’m trying to decode 160 character strings, 20 at a time, so using this whole library seems wrong… maybe I’ll do something like rabio suggests?

]]> By: rabio http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-830 rabio Wed, 30 Jul 2008 16:56:45 +0000 http://channel3b.wordpress.com/2007/07/04/how-to-convert-html-entities-to-real-unicode-in-python/#comment-830 Another great solution that not even require any external modules can be found on: http://effbot.org/zone/re-sub.htm#unescape-html Another great solution that not even require any external modules can
be found on: http://effbot.org/zone/re-sub.htm#unescape-html

]]>