utf.7 (2338B)
1 .deEX 2 .ift .ft5 3 .nf 4 .. 5 .deEE 6 .ft1 7 .fi 8 .. 9 .TH UTF 7 10 .SH NAME 11 UTF, Unicode, ASCII, rune \- character set and format 12 .SH DESCRIPTION 13 The Plan 9 character set and representation are 14 based on the Unicode Standard and on the ISO multibyte 15 .SM UTF-8 16 encoding (Universal Character 17 Set Transformation Format, 8 bits wide). 18 The Unicode Standard represents its characters in 16 19 bits; 20 .SM UTF-8 21 represents such 22 values in an 8-bit byte stream. 23 Throughout this manual, 24 .SM UTF-8 25 is shortened to 26 .SM UTF. 27 .PP 28 In Plan 9, a 29 .I rune 30 is a 16-bit quantity representing a Unicode character. 31 Internally, programs may store characters as runes. 32 However, any external manifestation of textual information, 33 in files or at the interface between programs, uses a 34 machine-independent, byte-stream encoding called 35 .SM UTF. 36 .PP 37 .SM UTF 38 is designed so the 7-bit 39 .SM ASCII 40 set (values hexadecimal 00 to 7F), 41 appear only as themselves 42 in the encoding. 43 Runes with values above 7F appear as sequences of two or more 44 bytes with values only from 80 to FF. 45 .PP 46 The 47 .SM UTF 48 encoding of the Unicode Standard is backward compatible with 49 .SM ASCII\c 50 : 51 programs presented only with 52 .SM ASCII 53 work on Plan 9 54 even if not written to deal with 55 .SM UTF, 56 as do 57 programs that deal with uninterpreted byte streams. 58 However, programs that perform semantic processing on 59 .SM ASCII 60 graphic 61 characters must convert from 62 .SM UTF 63 to runes 64 in order to work properly with non-\c 65 .SM ASCII 66 input. 67 See 68 .IR rune (3). 69 .PP 70 Letting numbers be binary, 71 a rune x is converted to a multibyte 72 .SM UTF 73 sequence 74 as follows: 75 .PP 76 01. x in [00000000.0bbbbbbb] → 0bbbbbbb 77 .br 78 10. x in [00000bbb.bbbbbbbb] → 110bbbbb, 10bbbbbb 79 .br 80 11. x in [bbbbbbbb.bbbbbbbb] → 1110bbbb, 10bbbbbb, 10bbbbbb 81 .br 82 .PP 83 Conversion 01 provides a one-byte sequence that spans the 84 .SM ASCII 85 character set in a compatible way. 86 Conversions 10 and 11 represent higher-valued characters 87 as sequences of two or three bytes with the high bit set. 88 Plan 9 does not support the 4, 5, and 6 byte sequences proposed by X-Open. 89 When there are multiple ways to encode a value, for example rune 0, 90 the shortest encoding is used. 91 .PP 92 In the inverse mapping, 93 any sequence except those described above 94 is incorrect and is converted to rune hexadecimal 0080. 95 .SH "SEE ALSO" 96 .IR ascii (1), 97 .IR tcs (1), 98 .IR rune (3), 99 .IR "The Unicode Standard" .