Hello,

You might know that the following ES expressions are broken:

	text.charAt(0) // get the first character of the text
	text.length > 100 ? text.substring(0,100) + '...' : text // cut the text after 100 characters

The reason is *not* because ES works with UTF-16 code units instead of Unicode code points (it's just a red herring!), but
because _graphemes_ (that is, what a human perceives as a "character") may span multiple code units and/or code points.
For example, the letter "n̈" ("n" with diaeresis) is coded using two code points, namely 
U+006E (LATIN SMALL LETTER N) and U+0308 (COMBINING DIAERESIS), and two UTF-16 code units, 
and it should be avoided to cut a string between these two codes in order to keep the resulting text meaningful for humans.
(Nota bene: I have carefully chosen a grapheme that exists in a current written language and does not exist 
as precomposed Unicode character.)

The correct technical notion to use here, is the notion of "grapheme cluster", that is 
a sequence of Unicode code points that represents a grapheme.
See [UAX29] (Unicode Standard Annex #29: Unicode text segmentation), section 3, for more info.

Therefore, I propose the following basic operations to operate on grapheme clusters:

(1) String.prototype.graphemeAt(pos)
	
This method is similar to `String.prototype.charAt()`, but it returns a grapheme cluster instead of a string 
composed of a single UTF-16 code unit. More precisely, it returns the shortest substring of `this` 
beginning at position `pos` (inclusively) and ending at position `pos2` (exclusively), where `pos2` is the 
smallest position in `this` which is greater than `pos` and which is an (extended) grapheme cluster boundary,
according to the specification in [UAX29], section 3.1.
If `pos` is out of bounds, an empty string is returned.

(2) String.prototype.graphemes(start = 0)

This method returns an iterator, enumerating the graphemes of `this`, starting at position `start`. 
Given the `String.prototype.graphemeAt` method as above,
it could be approximatively expressed in ES6 as follows (ignoring edge cases):

	String.prototype.graphemes = function*(pos = 0) {
		pos = Math.floor(pos)
		if (pos < 0 || Number.isNaN(pos))
			pos = 0
		while (pos < this.length) {
			let grapheme = this.graphemeAt(pos)
			pos += grapheme.length
			yield grapheme
		}
	}

So, the two examples of the beginning of my message could be correctly implemented as follows:

	text.graphemeAt(0) // get the first grapheme of the text

	// shorten a text to its first hundred graphemes
	var shortenText = ''
	let numGraphemes = 0
	for (let grapheme of text) {
		numGraphemes += 1
		if (numGraphemes > 100) {
			shortenText += '…'
			break
		}
		shortenText += grapheme
	}

As a side note, I ask whether the `String.prototype.symbolAt `/`String.prototype.at` as proposed in a recent thread, 
and the `String.prototype[@@iterator]` as currently specified, are really what people need, 
or if they would mistakenly use them with the intended meaning of `String.prototype.graphemeAt`
and `String.prototype.graphemes` as discussed in the present message?

Thoughts?

Claude

[UAX29]: http://www.unicode.org/reports/tr29/ "Unicode Standard Annex #29: Unicode text segmentation."

You might know that the following ES expressions are broken:

	text.charAt(0) // get the first character of the text
	text.length > 100 ? text.substring(0,100) + '...' : text // cut the text after 100 characters

The reason is *not* because ES works with UTF-16 code units instead of Unicode code points (it's just a red herring!), but
because _graphemes_ (that is, what a human perceives as a "character") may span multiple code units and/or code points.
For example, the letter "n̈" ("n" with diaeresis) is coded using two code points, namely 
U+006E (LATIN SMALL LETTER N) and U+0308 (COMBINING DIAERESIS), and two UTF-16 code units, 
and it should be avoided to cut a string between these two codes in order to keep the resulting text meaningful for humans.
(Nota bene: I have carefully chosen a grapheme that exists in a current written language and does not exist 
as precomposed Unicode character.)

The correct technical notion to use here, is the notion of "grapheme cluster", that is 
a sequence of Unicode code points that represents a grapheme.
See [UAX29] (Unicode Standard Annex #29: Unicode text segmentation), section 3, for more info.

Therefore, I propose the following basic operations to operate on grapheme clusters:

(1) String.prototype.graphemeAt(pos)
	
This method is similar to `String.prototype.charAt()`, but it returns a grapheme cluster instead of a string 
composed of a single UTF-16 code unit. More precisely, it returns the shortest substring of `this` 
beginning at position `pos` (inclusively) and ending at position `pos2` (exclusively), where `pos2` is the 
smallest position in `this` which is greater than `pos` and which is an (extended) grapheme cluster boundary,
according to the specification in [UAX29], section 3.1.
If `pos` is out of bounds, an empty string is returned.

(2) String.prototype.graphemes(start = 0)

This method returns an iterator, enumerating the graphemes of `this`, starting at position `start`. 
Given the `String.prototype.graphemeAt` method as above,
it could be approximatively expressed in ES6 as follows (ignoring edge cases):

	String.prototype.graphemes = function*(pos = 0) {
		pos = Math.floor(pos)
		if (pos < 0 || Number.isNaN(pos))
			pos = 0
		while (pos < this.length) {
			let grapheme = this.graphemeAt(pos)
			pos += grapheme.length
			yield grapheme
		}
	}

So, the two examples of the beginning of my message could be correctly implemented as follows:

	text.graphemeAt(0) // get the first grapheme of the text

	// shorten a text to its first hundred graphemes
	var shortenText = ''
	let numGraphemes = 0
	for (let grapheme of text) {
		numGraphemes += 1
		if (numGraphemes > 100) {
			shortenText += '…'
			break
		}
		shortenText += grapheme
	}

As a side note, I ask whether the `String.prototype.symbolAt `/`String.prototype.at` as proposed in a recent thread, 
and the `String.prototype[@@iterator]` as currently specified, are really what people need, 
or if they would mistakenly use them with the intended meaning of `String.prototype.graphemeAt`
and `String.prototype.graphemes` as discussed in the present message?

Thoughts?

[UAX29]: http://www.unicode.org/reports/tr29/ "Unicode Standard Annex #29: Unicode text segmentation."