{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "handled-recruitment",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'6.0.3'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import ftfy\n",
    "ftfy.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eleven-caution",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Mojibake!\n",
    "\n",
    "## What the hâ€”ck happened to this text?\n",
    "\n",
    "Robyn Speer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "confused-advertiser",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Mojibake is when text ends up with the wrong Unicode characters due to an encoding mistake.\n",
    "\n",
    "- It's Japanese for \"ghost characters\"\n",
    "- ftfy is my Python library that fixes them. (get it with `pip install ftfy`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "disabled-ridge",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'merci de télécharger le plug-in'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ftfy.fix_text(\"merci de t‚Äö√†√∂¬¨¬©l‚Äö√†√∂¬¨¬©charger le plug-in\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "focused-facial",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The mascot of ftfy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "introductory-supervisor",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"(ง'̀⌣'́)ง\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ftfy.fix_text(\"(Ã\\xa0Â¸â€¡'ÃŒâ‚¬Ã¢Å’Â£'ÃŒÂ\\x81)Ã\\xa0Â¸â€¡\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "following-apple",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "This little guy, and in fact every example in this talk, comes from mojibake I found in the wild -- usually on Twitter or in the OSCAR Web Corpus.\n",
    "\n",
    "> **Side note**: there are a lot of tangents I would like to go off on, but this is a 10-minute talk and there's no time for tangents. So I'll be sprinkling in these side notes that I'll be skipping over as I present the talk.\n",
    ">\n",
    "> If you're interested in them, I suggest pausing the video, or reading the Jupyter Notebook version of this talk that I'll be linking later."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "concrete-consent",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It used to be that 1 byte = 1 character, so there were at most 256 possible characters of text that could be shown on a computer. Here's some quick code to see them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "horizontal-playlist",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import blessings\n",
    "term = blessings.Terminal()  # enable colorful text\n",
    "\n",
    "def displayable_codepoint(codepoint, encoding):\n",
    "    char = bytes([codepoint]).decode(encoding, 'replace')\n",
    "    if char == '�':\n",
    "        return '▓▓'\n",
    "    elif not char.isprintable():\n",
    "        return '░░'\n",
    "    else:\n",
    "        return char\n",
    "\n",
    "def show_char_table(encoding):\n",
    "    print(f\"encoding: {encoding}\\n     0 1 2 3 4 5 6 7 8 9 a b c d e f\\n\")\n",
    "    for row in range(16):\n",
    "        print(f\"{row*16:>02x}\", end=\"   \")\n",
    "        if row == 0:\n",
    "            print(ftfy.formatting.display_center(term.green(\" control characters \"), 32, \"░\"))\n",
    "        elif row == 8 and encoding == 'latin-1':\n",
    "            print(ftfy.formatting.display_center(term.green(\" here be dragons \"), 32, \"░\"))\n",
    "        else:\n",
    "            for col in range(16):\n",
    "                char = displayable_codepoint(row * 16 + col, encoding)\n",
    "                print(f\"{char:<2}\", end=\"\")\n",
    "            print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "worthy-broad",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ASCII\n",
    "\n",
    "- In the '60s, we agreed that 128 of these bytes should have well-defined meanings as characters of text. That's ASCII\n",
    "- It worked pretty well for monolingual Americans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "interim-attribute",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoding: ascii\n",
      "     0 1 2 3 4 5 6 7 8 9 a b c d e f\n",
      "\n",
      "00   ░░░░░░\u001b[32m control characters \u001b[m░░░░░░\n",
      "10   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n",
      "20     ! \" # $ % & ' ( ) * + , - . / \n",
      "30   0 1 2 3 4 5 6 7 8 9 : ; < = > ? \n",
      "40   @ A B C D E F G H I J K L M N O \n",
      "50   P Q R S T U V W X Y Z [ \\ ] ^ _ \n",
      "60   ` a b c d e f g h i j k l m n o \n",
      "70   p q r s t u v w x y z { | } ~ ░░\n",
      "80   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓\n",
      "90   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓\n",
      "a0   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓\n",
      "b0   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓\n",
      "c0   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓\n",
      "d0   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓\n",
      "e0   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓\n",
      "f0   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓\n"
     ]
    }
   ],
   "source": [
    "show_char_table(\"ascii\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "starting-velvet",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- Basically everyone decided the other 128 bytes should be characters too\n",
    "- So people started talking about \"extended ASCII\" which means \"whatever my computer does with the other bytes\"\n",
    "- A different computer, in a different country or with a different OS, would do something different\n",
    "- This is how mojibake started\n",
    "\n",
    "Let's take a look at the different ways that ASCII got \"extended\", which are now called _codepages_."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "flying-oriental",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here's Latin-1. This encoding was used on a lot of UNIX-like systems before they switched to UTF-8, but probably the biggest reason you see it is that it's the first 256 characters of Unicode.\n",
    "\n",
    "If you don't know about decoding text and you just replace each byte with the Unicode character with the same number, Latin-1 is the encoding you get by accident."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "european-ethics",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoding: latin-1\n",
      "     0 1 2 3 4 5 6 7 8 9 a b c d e f\n",
      "\n",
      "00   ░░░░░░\u001b[32m control characters \u001b[m░░░░░░\n",
      "10   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n",
      "20     ! \" # $ % & ' ( ) * + , - . / \n",
      "30   0 1 2 3 4 5 6 7 8 9 : ; < = > ? \n",
      "40   @ A B C D E F G H I J K L M N O \n",
      "50   P Q R S T U V W X Y Z [ \\ ] ^ _ \n",
      "60   ` a b c d e f g h i j k l m n o \n",
      "70   p q r s t u v w x y z { | } ~ ░░\n",
      "80   ░░░░░░░\u001b[32m here be dragons \u001b[m░░░░░░░░\n",
      "90   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n",
      "a0   ░░¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ░░® ¯ \n",
      "b0   ° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿ \n",
      "c0   À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï \n",
      "d0   Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß \n",
      "e0   à á â ã ä å æ ç è é ê ë ì í î ï \n",
      "f0   ð ñ ò ó ô õ ö ÷ ø ù ú û ü ý þ ÿ \n"
     ]
    }
   ],
   "source": [
    "\n",
    "show_char_table('latin-1')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "editorial-clinton",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Dragons? 🐉\n",
    "\n",
    "I labeled rows 80 and 90 as \"here be dragons\" because they're full of control characters, but nobody ever agreed on what they do.\n",
    "\n",
    "Text that includes these codepoints can be shown as nearly anything, including little boxes, characters from the Windows-1252 character set, or just messing up your whole terminal window.\n",
    "\n",
    "> Side note: Control character 85 was an attempt to resolve the war between Unix line breaks (`0a`), Windows line breaks (`0d 0a`), and Mac Classic line breaks (`0d`), by introducing an _entirely new_ line break that wasn't even in ASCII.\n",
    ">\n",
    "> This is hilarious in retrospect. Clearly nobody would ever repeat that terrible idea, except the Unicode Consortium, who did it twice more.\n",
    ">\n",
    "> Anyway, ftfy has a function that turns all of these into byte `0a`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "baking-notebook",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoding: windows-1252\n",
      "     0 1 2 3 4 5 6 7 8 9 a b c d e f\n",
      "\n",
      "00   ░░░░░░\u001b[32m control characters \u001b[m░░░░░░\n",
      "10   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n",
      "20     ! \" # $ % & ' ( ) * + , - . / \n",
      "30   0 1 2 3 4 5 6 7 8 9 : ; < = > ? \n",
      "40   @ A B C D E F G H I J K L M N O \n",
      "50   P Q R S T U V W X Y Z [ \\ ] ^ _ \n",
      "60   ` a b c d e f g h i j k l m n o \n",
      "70   p q r s t u v w x y z { | } ~ ░░\n",
      "80   € ▓▓‚ ƒ „ … † ‡ ˆ ‰ Š ‹ Œ ▓▓Ž ▓▓\n",
      "90   ▓▓‘ ’ “ ” • – — ˜ ™ š › œ ▓▓ž Ÿ \n",
      "a0   ░░¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ░░® ¯ \n",
      "b0   ° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿ \n",
      "c0   À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï \n",
      "d0   Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß \n",
      "e0   à á â ã ä å æ ç è é ê ë ì í î ï \n",
      "f0   ð ñ ò ó ô õ ö ÷ ø ù ú û ü ý þ ÿ \n"
     ]
    }
   ],
   "source": [
    "show_char_table('windows-1252')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "affiliated-gentleman",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoding: cp437\n",
      "     0 1 2 3 4 5 6 7 8 9 a b c d e f\n",
      "\n",
      "00   ░░░░░░\u001b[32m control characters \u001b[m░░░░░░\n",
      "10   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n",
      "20     ! \" # $ % & ' ( ) * + , - . / \n",
      "30   0 1 2 3 4 5 6 7 8 9 : ; < = > ? \n",
      "40   @ A B C D E F G H I J K L M N O \n",
      "50   P Q R S T U V W X Y Z [ \\ ] ^ _ \n",
      "60   ` a b c d e f g h i j k l m n o \n",
      "70   p q r s t u v w x y z { | } ~ ░░\n",
      "80   Ç ü é â ä à å ç ê ë è ï î ì Ä Å \n",
      "90   É æ Æ ô ö ò û ù ÿ Ö Ü ¢ £ ¥ ₧ ƒ \n",
      "a0   á í ó ú ñ Ñ ª º ¿ ⌐ ¬ ½ ¼ ¡ « » \n",
      "b0   ░ ▒ ▓ │ ┤ ╡ ╢ ╖ ╕ ╣ ║ ╗ ╝ ╜ ╛ ┐ \n",
      "c0   └ ┴ ┬ ├ ─ ┼ ╞ ╟ ╚ ╔ ╩ ╦ ╠ ═ ╬ ╧ \n",
      "d0   ╨ ╤ ╥ ╙ ╘ ╒ ╓ ╫ ╪ ┘ ┌ █ ▄ ▌ ▐ ▀ \n",
      "e0   α ß Γ π Σ σ µ τ Φ Θ Ω δ ∞ φ ε ∩ \n",
      "f0   ≡ ± ≥ ≤ ⌠ ⌡ ÷ ≈ ° ∙ · √ ⁿ ² ■ ░░\n"
     ]
    }
   ],
   "source": [
    "show_char_table('cp437')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "northern-string",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoding: macroman\n",
      "     0 1 2 3 4 5 6 7 8 9 a b c d e f\n",
      "\n",
      "00   ░░░░░░\u001b[32m control characters \u001b[m░░░░░░\n",
      "10   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n",
      "20     ! \" # $ % & ' ( ) * + , - . / \n",
      "30   0 1 2 3 4 5 6 7 8 9 : ; < = > ? \n",
      "40   @ A B C D E F G H I J K L M N O \n",
      "50   P Q R S T U V W X Y Z [ \\ ] ^ _ \n",
      "60   ` a b c d e f g h i j k l m n o \n",
      "70   p q r s t u v w x y z { | } ~ ░░\n",
      "80   Ä Å Ç É Ñ Ö Ü á à â ä ã å ç é è \n",
      "90   ê ë í ì î ï ñ ó ò ô ö õ ú ù û ü \n",
      "a0   † ° ¢ £ § • ¶ ß ® © ™ ´ ¨ ≠ Æ Ø \n",
      "b0   ∞ ± ≤ ≥ ¥ µ ∂ ∑ ∏ π ∫ ª º Ω æ ø \n",
      "c0   ¿ ¡ ¬ √ ƒ ≈ ∆ « » … ░░À Ã Õ Œ œ \n",
      "d0   – — “ ” ‘ ’ ÷ ◊ ÿ Ÿ ⁄ € ‹ › ﬁ ﬂ \n",
      "e0   ‡ · ‚ „ ‰ Â Ê Á Ë È Í Î Ï Ì Ó Ô \n",
      "f0   ░░Ò Ú Û Ù ı ˆ ˜ ¯ ˘ ˙ ˚ ¸ ˝ ˛ ˇ \n"
     ]
    }
   ],
   "source": [
    "show_char_table('macroman')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "intelligent-equipment",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoding: windows-1251\n",
      "     0 1 2 3 4 5 6 7 8 9 a b c d e f\n",
      "\n",
      "00   ░░░░░░\u001b[32m control characters \u001b[m░░░░░░\n",
      "10   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n",
      "20     ! \" # $ % & ' ( ) * + , - . / \n",
      "30   0 1 2 3 4 5 6 7 8 9 : ; < = > ? \n",
      "40   @ A B C D E F G H I J K L M N O \n",
      "50   P Q R S T U V W X Y Z [ \\ ] ^ _ \n",
      "60   ` a b c d e f g h i j k l m n o \n",
      "70   p q r s t u v w x y z { | } ~ ░░\n",
      "80   Ђ Ѓ ‚ ѓ „ … † ‡ € ‰ Љ ‹ Њ Ќ Ћ Џ \n",
      "90   ђ ‘ ’ “ ” • – — ▓▓™ љ › њ ќ ћ џ \n",
      "a0   ░░Ў ў Ј ¤ Ґ ¦ § Ё © Є « ¬ ░░® Ї \n",
      "b0   ° ± І і ґ µ ¶ · ё № є » ј Ѕ ѕ ї \n",
      "c0   А Б В Г Д Е Ж З И Й К Л М Н О П \n",
      "d0   Р С Т У Ф Х Ц Ч Ш Щ Ъ Ы Ь Э Ю Я \n",
      "e0   а б в г д е ж з и й к л м н о п \n",
      "f0   р с т у ф х ц ч ш щ ъ ы ь э ю я \n"
     ]
    }
   ],
   "source": [
    "show_char_table(\"windows-1251\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unique-airplane",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## So text wasn't portable between different computers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "compound-columbia",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Plus Áa change, plus cíest la mÍme chose'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "phrase = \"Plus ça change, plus c’est la même chose\"\n",
    "\n",
    "phrase.encode('windows-1252').decode('macroman')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "identical-welsh",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Vintage '90s mojibake\n",
    "\n",
    "Maybe you've seen a \".NFO\" file like this, telling you about the nice people who provided you with a DOS game for free:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "scenic-norman",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "crack_nfo = r\"\"\"\n",
    "  ───── ────────────── ───────────────── ────────────── ─────────────── ────\n",
    " ▄█████▄ ▀█████▄ ████▄ ▀█████████ ████▄   ▄█████▄▀████▄ ▀██████▄ ▀████▄  ▄██▄\n",
    " ████████ █████▀ ██████ ▀████████ ██████ ▀███████▌██████  ███████  ████▄█████\n",
    " ███ ▀███▌█▀ ▄▄▄█▌▀█████ ▌████ ▄▄▄█▀▐████ ███▀▌▀█▌█▌ ███▌ ██▀▐████ ██████████\n",
    " ███  ▐██▌▌  ████  ▌████  ████ ████  ████ ██▌  ▌▐▌█▌ ████ ██  ████ ██▌▐█▌▐███\n",
    " ███ ▄███▌▄▄ ████   ████  ████ ████ ▄████ ██▌  █▄▄█▌ ████ ██ ▄████ ██ ▐█▌ ███\n",
    " ████████ ██ ████   ████  ████ ████ █████ ██▌  ▀▀██▌▐███▀ ██▐█████ ██  █  ███\n",
    " ██████▀ ▄▀▀ ████   ████  ████ ████ ▀████ ██▌███▄▐██▄▀▀ ▄███ ▀████ ██  ▄  ███\n",
    " ███▀ ▄▄██   ████  ▐████  ████ ████  ████ ██▌▐▐██▌█▀██▄ ████  ████ ██     ███\n",
    " ███  █████▄  ▀▀█  ████▀ ▐████ ░███ ▐████ ███▄▐██▌█▌▐██▌▐███ ▐███░ ██▌   r███\n",
    " ░██  █████████▄   ▐██▌▄██████ ▒░██ █████ ███████▌█▌▐███ ███ ███░▒ ███   o██░\n",
    " ▒░█   ▀███████▀  ▀ ███▐████▀  ▓▒░█ ▐███▌ ▀██████▐█▌▐███ ███ ▐█░▒▓ ██▌   y█░▒\n",
    " - ▌─────▐▀─ ▄▄▄█ ── ▀▀ ───── ────── ▀▀▀ ─ ▐▀▀▀▀ ▀▀ ████ ──── ▀█▀ ─ ▀ ────▐ ─\n",
    "\n",
    "   ╓────────────────────────[ RELEASE INFORMATION ]───────────────────────╖\n",
    "╓────────────────────────────────────────────────────────────────────────────╖\n",
    "║  -/\\- THE EVEN MORE INCREDIBLE MACHINE FOR *DOS* FROM SIERRA/DYNAMIX -/\\-  ║\n",
    "╙────────────────────────────────────────────────────────────────────────────╜\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "graphic-treasurer",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "> **Side note**: It's funny that most of the things we call \"ASCII art\" aren't ASCII."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "valuable-ethernet",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "  ÄÄÄÄÄ ÄÄÄÄÄÄÄÄÄÄÄÄÄÄ ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ ÄÄÄÄÄÄÄÄÄÄÄÄÄÄ ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ ÄÄÄÄ\n",
      " ÜÛÛÛÛÛÜ ßÛÛÛÛÛÜ ÛÛÛÛÜ ßÛÛÛÛÛÛÛÛÛ ÛÛÛÛÜ   ÜÛÛÛÛÛÜßÛÛÛÛÜ ßÛÛÛÛÛÛÜ ßÛÛÛÛÜ  ÜÛÛÜ\n",
      " ÛÛÛÛÛÛÛÛ ÛÛÛÛÛß ÛÛÛÛÛÛ ßÛÛÛÛÛÛÛÛ ÛÛÛÛÛÛ ßÛÛÛÛÛÛÛÝÛÛÛÛÛÛ  ÛÛÛÛÛÛÛ  ÛÛÛÛÜÛÛÛÛÛ\n",
      " ÛÛÛ ßÛÛÛÝÛß ÜÜÜÛÝßÛÛÛÛÛ ÝÛÛÛÛ ÜÜÜÛßÞÛÛÛÛ ÛÛÛßÝßÛÝÛÝ ÛÛÛÝ ÛÛßÞÛÛÛÛ ÛÛÛÛÛÛÛÛÛÛ\n",
      " ÛÛÛ  ÞÛÛÝÝ  ÛÛÛÛ  ÝÛÛÛÛ  ÛÛÛÛ ÛÛÛÛ  ÛÛÛÛ ÛÛÝ  ÝÞÝÛÝ ÛÛÛÛ ÛÛ  ÛÛÛÛ ÛÛÝÞÛÝÞÛÛÛ\n",
      " ÛÛÛ ÜÛÛÛÝÜÜ ÛÛÛÛ   ÛÛÛÛ  ÛÛÛÛ ÛÛÛÛ ÜÛÛÛÛ ÛÛÝ  ÛÜÜÛÝ ÛÛÛÛ ÛÛ ÜÛÛÛÛ ÛÛ ÞÛÝ ÛÛÛ\n",
      " ÛÛÛÛÛÛÛÛ ÛÛ ÛÛÛÛ   ÛÛÛÛ  ÛÛÛÛ ÛÛÛÛ ÛÛÛÛÛ ÛÛÝ  ßßÛÛÝÞÛÛÛß ÛÛÞÛÛÛÛÛ ÛÛ  Û  ÛÛÛ\n",
      " ÛÛÛÛÛÛß Üßß ÛÛÛÛ   ÛÛÛÛ  ÛÛÛÛ ÛÛÛÛ ßÛÛÛÛ ÛÛÝÛÛÛÜÞÛÛÜßß ÜÛÛÛ ßÛÛÛÛ ÛÛ  Ü  ÛÛÛ\n",
      " ÛÛÛß ÜÜÛÛ   ÛÛÛÛ  ÞÛÛÛÛ  ÛÛÛÛ ÛÛÛÛ  ÛÛÛÛ ÛÛÝÞÞÛÛÝÛßÛÛÜ ÛÛÛÛ  ÛÛÛÛ ÛÛ     ÛÛÛ\n",
      " ÛÛÛ  ÛÛÛÛÛÜ  ßßÛ  ÛÛÛÛß ÞÛÛÛÛ °ÛÛÛ ÞÛÛÛÛ ÛÛÛÜÞÛÛÝÛÝÞÛÛÝÞÛÛÛ ÞÛÛÛ° ÛÛÝ   rÛÛÛ\n",
      " °ÛÛ  ÛÛÛÛÛÛÛÛÛÜ   ÞÛÛÝÜÛÛÛÛÛÛ ±°ÛÛ ÛÛÛÛÛ ÛÛÛÛÛÛÛÝÛÝÞÛÛÛ ÛÛÛ ÛÛÛ°± ÛÛÛ   oÛÛ°\n",
      " ±°Û   ßÛÛÛÛÛÛÛß  ß ÛÛÛÞÛÛÛÛß  ²±°Û ÞÛÛÛÝ ßÛÛÛÛÛÛÞÛÝÞÛÛÛ ÛÛÛ ÞÛ°±² ÛÛÝ   yÛ°±\n",
      " - ÝÄÄÄÄÄÞßÄ ÜÜÜÛ ÄÄ ßß ÄÄÄÄÄ ÄÄÄÄÄÄ ßßß Ä Þßßßß ßß ÛÛÛÛ ÄÄÄÄ ßÛß Ä ß ÄÄÄÄÞ Ä\n",
      "\n",
      "   ÖÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ[ RELEASE INFORMATION ]ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ·\n",
      "ÖÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ·\n",
      "º  -/\\- THE EVEN MORE INCREDIBLE MACHINE FOR *DOS* FROM SIERRA/DYNAMIX -/\\-  º\n",
      "ÓÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ½\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(crack_nfo.encode('cp437').decode('windows-1252'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alpha-ticket",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Here comes UTF-8\n",
    "\n",
    "Instead of using the other 128 bytes for 128 more characters, what if we used them as a variable-length encoding for the whole rest of Unicode?\n",
    "\n",
    "There were other variable-length encodings, but UTF-8 is well-designed.\n",
    "\n",
    "- It leaves ASCII as ASCII\n",
    "- It doesn't overlap with ASCII _ever_: ASCII bytes always exclusively stand for the characters you expect\n",
    "- It's self-synchronizing, so you can tell where each character starts and ends even with no context\n",
    "\n",
    "Everyone recognized how good this idea was and switched every system to UTF-8. Encoding problems were solved forever, the end.\n",
    "\n",
    "I'm kidding. UTF-8 is great but not everyone adopted it, especially not Microsoft who had just done _all that work_ to switch Windows APIs to UTF-16, which nobody likes. So now we have more kinds of mojibake."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "textile-abuse",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "l = 6c\n",
      "’ = e2 80 99\n",
      "H = 48\n",
      "ô = c3 b4\n",
      "p = 70\n",
      "i = 69\n",
      "t = 74\n",
      "a = 61\n",
      "l = 6c\n"
     ]
    }
   ],
   "source": [
    "# Code to look at the encoding of each character in UTF-8\n",
    "\n",
    "def show_utf8(text):\n",
    "    for char in text:\n",
    "        char_bytes = char.encode('utf-8')\n",
    "        byte_sequence = ' '.join([f\"{byte:>02x}\" for byte in char_bytes])\n",
    "        print(f\"{char} = {byte_sequence}\")\n",
    "\n",
    "text = \"l’Hôpital\"\n",
    "show_utf8(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "worse-calculation",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What happens when not everyone is on board with UTF-8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "adjustable-northeast",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'l\\xe2\\x80\\x99H\\xc3\\xb4pital'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text.encode('utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "printable-practitioner",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lâ€™HÃ´pital\n"
     ]
    }
   ],
   "source": [
    "print(text.encode('utf-8').decode('windows-1252'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "eleven-radius",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lÃ¢â‚¬â„¢HÃƒÂ´pital\n"
     ]
    }
   ],
   "source": [
    "print(text.encode('utf-8').decode('windows-1252').encode('utf-8').decode('windows-1252'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "sophisticated-volunteer",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "EXAMPLES = [\n",
    "    \"Merci de t‚Äö√†√∂¬¨¬©l‚Äö√†√∂¬¨¬©charger le plug-in\",\n",
    "    \n",
    "    \"The Mona Lisa doesnÃƒÂ¢Ã¢â€šÂ¬Ã¢â€žÂ¢t have eyebrows.\",\n",
    "    \n",
    "    \"I just figured out how to tweet emojis! â\\x9a½í\\xa0½í¸\\x80í\\xa0½í¸\\x81í\\xa0½í¸\"\n",
    "    \"\\x82í\\xa0½í¸\\x86í\\xa0½í¸\\x8eí\\xa0½í¸\\x8eí\\xa0½í¸\\x8eí\\xa0½í¸\\x8e\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "surrounded-makeup",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## We can recognize UTF-8 mojibake by its distinct patterns\n",
    "\n",
    "The pattern of bytes that makes UTF-8 self-synchronizing is a pattern we can recognize even via a different encoding.\n",
    "\n",
    "The example `doesnÃƒÂ¢Ã¢â€šÂ¬Ã¢â€žÂ¢t` is recognizable as UTF-8 / Windows-1252 mojibake, for example. `t‚Äö√†√∂¬¨¬©l‚Äö√†√∂¬¨¬©charger` is recognizable as UTF-8 / MacRoman mojibake.\n",
    "\n",
    "When we see such a pattern, we encode as the appropriate other encoding, then decode as UTF-8."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "sunrise-engineering",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Merci de t‚àö¬©l‚àö¬©charger le plug-in'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "EXAMPLES[0].encode('macroman').decode('utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "portuguese-marble",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Merci de t√©l√©charger le plug-in'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_.encode('macroman').decode('utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "angry-floor",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Merci de télécharger le plug-in'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_.encode('macroman').decode('utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "historic-cameroon",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## This is a job that the computer can do for us"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "sunrise-madagascar",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Some code to format the output of ftfy\n",
    "\n",
    "from ftfy import fix_and_explain\n",
    "from pprint import pprint\n",
    "\n",
    "def show_explanation(text):\n",
    "    print(f\"Original: {text}\")\n",
    "    fixed, expl = fix_and_explain(text)\n",
    "    print(f\"   Fixed: {fixed}\\n\")\n",
    "    pprint(expl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "adjusted-civilization",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original: Merci de t‚Äö√†√∂¬¨¬©l‚Äö√†√∂¬¨¬©charger le plug-in\n",
      "   Fixed: Merci de télécharger le plug-in\n",
      "\n",
      "[('encode', 'macroman'),\n",
      " ('decode', 'utf-8'),\n",
      " ('encode', 'macroman'),\n",
      " ('decode', 'utf-8'),\n",
      " ('encode', 'macroman'),\n",
      " ('decode', 'utf-8')]\n"
     ]
    }
   ],
   "source": [
    "show_explanation(EXAMPLES[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "attended-soviet",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original: The Mona Lisa doesnÃƒÂ¢Ã¢â€šÂ¬Ã¢â€žÂ¢t have eyebrows.\n",
      "   Fixed: The Mona Lisa doesn't have eyebrows.\n",
      "\n",
      "[('encode', 'sloppy-windows-1252'),\n",
      " ('decode', 'utf-8'),\n",
      " ('encode', 'sloppy-windows-1252'),\n",
      " ('decode', 'utf-8'),\n",
      " ('encode', 'sloppy-windows-1252'),\n",
      " ('decode', 'utf-8'),\n",
      " ('apply', 'uncurl_quotes')]\n"
     ]
    }
   ],
   "source": [
    "show_explanation(EXAMPLES[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "yellow-running",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original: I just figured out how to tweet emojis! â½í ½í¸í ½í¸í ½í¸í ½í¸í ½í¸í ½í¸í ½í¸í ½í¸\n",
      "   Fixed: I just figured out how to tweet emojis! ⚽😀😁😂😆😎😎😎😎\n",
      "\n",
      "[('encode', 'latin-1'), ('decode', 'utf-8-variants')]\n"
     ]
    }
   ],
   "source": [
    "show_explanation(EXAMPLES[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "laden-trick",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "> **Side note**: ftfy adds encodings to Python like `sloppy-windows-1252` and `utf-8-variants` when it's imported. Python is very strict about encoding standards, but to deal with real mojibake, we have to be very loose about them. For example, this tweet requires us to recognize and simulate a broken implementation of UTF-8."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "indirect-grounds",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Avoiding false positives\n",
    "\n",
    "ftfy only changes text that trips its mojibake-detector regex.\n",
    "\n",
    "Here are some examples that ftfy _could_ consider to be UTF-8 mojibake and try to \"fix\", but thankfully it doesn't:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "specific-buddy",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Con il corpo e lo spirito ammaccato,ʏ come se nel cuore avessi un vetro conficcato.\n",
      "2012Ѱ\n",
      "TEM QUE SEGUIR, SDV SӅ\n",
      "Join ZZAJɒs Official Fan List\n",
      "(-1/2)! = ù\n",
      "OK??:(   `«    ):\n"
     ]
    }
   ],
   "source": [
    "NEGATIVE_EXAMPLES = [\n",
    "    \"Con il corpo e lo spirito ammaccato,\\u00a0è come se nel cuore avessi un vetro conficcato.\",\n",
    "    \"2012—∞\",\n",
    "    \"TEM QUE SEGUIR, SDV SÓ…\",\n",
    "    \"Join ZZAJÉ’s Official Fan List\",\n",
    "    \"(-1/2)! = √π\",\n",
    "    \"OK??:(   `¬´    ):\"\n",
    "]\n",
    "\n",
    "for example in NEGATIVE_EXAMPLES:\n",
    "    # ftfy doesn't \"fix\" these because they're not broken, but we can manually try fixes\n",
    "    try:\n",
    "        print(example.encode('sloppy-windows-1252').decode('utf-8'))\n",
    "    except UnicodeError:\n",
    "        print(example.encode('macroman').decode('utf-8'))\n",
    "    assert ftfy.fix_encoding(example) == example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prompt-playing",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The \"badness\" metric\n",
    "\n",
    "ftfy doesn't just look for UTF-8-like patterns of characters, it also makes sure they are unlikely to be the intended text.\n",
    "\n",
    "- Improbable combinations: accented letters next to currency signs, math symbols next to console line art\n",
    "- Lots of things involving capital Ã and Â, where it doesn't look like they're being used for real in a capitalized word\n",
    "- Unloved characters like `¶` PILCROW SIGN, `‡` DOUBLE DAGGER, `◊` LOZENGE next to other mojibake-related characters\n",
    "\n",
    "...and many more cases that it looks for in a big regex.\n",
    "\n",
    "Strings that match the regex can be re-decoded. Specific character sequences that match the regex can be reinterpreted even if they're inconsistent with the rest of the string.\n",
    "\n",
    "> **Side note**: We used to try to categorize every Unicode character to find \"badness\". Now we only categorize the 400 or so characters that actually can appear in UTF-8 mojibake, because ftfy wouldn't have a reason to replace the other characters anyway."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "reserved-sheriff",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "text = \"Ã perturber la rÃ©flexion des thÃ©ologiens jusqu'Ã nos jours\"\n",
    "\n",
    "# We want to highlight the matches to this regular expression:\n",
    "ftfy.badness.BADNESS_RE.findall(text)\n",
    "\n",
    "# We'll just highlight it manually:\n",
    "term = blessings.Terminal()\n",
    "highlighted_text = term.on_yellow(\"Ã \") + \"perturber la r\" + term.on_yellow(\"Ã©\") + \"flexion des th\" + term.on_yellow(\"Ã©\") + \"ologiens jusqu'Ã nos jours\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "developing-intervention",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[43mÃ \u001b[mperturber la r\u001b[43mÃ©\u001b[mflexion des th\u001b[43mÃ©\u001b[mologiens jusqu'Ã nos jours\n",
      "à perturber la réflexion des théologiens jusqu'à nos jours\n"
     ]
    }
   ],
   "source": [
    "# Highlighted text shows matches for the 'badness' expression.\n",
    "# If we've confirmed from them that this is mojibake, and there's a consistent fix, we\n",
    "# can fix even text in contexts that were too unclear for the regex, such as the final Ã.\n",
    "\n",
    "print(highlighted_text)\n",
    "print(ftfy.fix_text(highlighted_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "suspended-angel",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ftfy is a hand-tuned heuristic. Why doesn't it use machine learning?\n",
    "\n",
    "I don't want ftfy to have false positives. It does, but every one of them is a bug I should fix. The actual rate of false positives should be once in several gigabytes of natural text.\n",
    "\n",
    "- Machine learning techniques aren't designed to have error rates this low\n",
    "- Machine learning would have a tendency to make its output look like what a language model \"expects\" even if the text didn't say that"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fourth-pepper",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why does mojibake keep happening?\n",
    "\n",
    "The top 3 root causes:\n",
    "\n",
    "1. Microsoft Excel\n",
    "2. Programming language APIs that let you confuse bytes and text\n",
    "3. An outdated heuristic called `chardet`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "outstanding-metropolitan",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## chardet\n",
    "\n",
    "`chardet` is a heuristic that takes in unlabeled bytes and tries to guess what encoding they're in. It was designed as part of Netscape Navigator in 1998, then ported to Python and other languages.\n",
    "\n",
    "It doesn't know that the correct answer to \"what encoding is this?\" is usually \"UTF-8\", and it thinks emoji are some kind of Turkish.\n",
    "\n",
    "> **Side note**: I recognize that we could be looking at ftfy the same way one day, particularly if I stop updating ftfy's heuristic. But chardet is fundamentally built on assumptions that aren't true anymore, and its original developer decided long ago that he'd written enough Python for a lifetime and he was going to do something else.\n",
    ">\n",
    "> It was an okay idea at the time, but we should have been able to replace it in two decades."
   ]
  },
  {
   "attachments": {
    "excel-export.png": {
     "image/png": "[encoded data, 25536 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "reduced-needle",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Microsoft Excel and the Three Decades of Backward Compatibility\n",
    "\n",
    "- Excel gives you lots of text formats you can export in, and _every one of them is wrong in 2021_\n",
    "- Even the one that says \"Unicode text\" is not what you want\n",
    "- The format of \"CSV\" depends on your OS, language, and region, like pre-Unicode days, and it will almost certainly mojibake your text\n",
    "\n",
    "I know you probably have to use Excel sometimes, but my recommendation is to make CSVs with LibreOffice or Google Spreadsheets.\n",
    "\n",
    "![excel-export.png](attachment:excel-export.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "legendary-catholic",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## How to avoid mojibake\n",
    "\n",
    "The bad news is that English speakers can go for a long time without noticing mojibake.\n",
    "\n",
    "The good news is that emoji are *everywhere*, people expect them to work, and they quickly reveal if you're doing Unicode right.\n",
    "\n",
    "- Use lots of emoji! 👍💚🌠\n",
    "- Use non-ASCII text, such as emoji, in your code, your UIs, your test cases\n",
    "- This is similar to how Web frameworks used to pass around parameters like `&snowman=☃` and `&utf8=✔` to make sure browsers used UTF-8 and detect if they came out wrong\n",
    "\n",
    "> **Side note**: Services such as Slack and Discord don't use Unicode for their emoji. They use ASCII strings like `:green-heart:` and turn them into images. These won't help you test anything. I recommend getting emoji for your test cases by copy-pasting them from emojipedia.org.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sticky-reminder",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Thanks!\n",
    "\n",
    "- ftfy documentation: https://ftfy.readthedocs.org\n",
    "- My e-mail address: rspeer@arborelia.net\n",
    "- I'm @r_speer on Twitter\n",
    "- I'll link from my Twitter to the notebook version of this talk\n",
    "- BTW, I'm on the job market right now\n",
    "\n",
    "Fonts I used for code in the presentation:\n",
    "\n",
    "- Input: Fantasque Sans Mono\n",
    "- Output: Fira Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "included-california",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
