Encoding and character sets
Handle non-ASCII text, +, and %20 in URLs
Tools to use
A search phrase loses spaces or turns plus signs into spaces even though you intended to send the same text. URL bugs depend not only on how many conversions occurred, but on what the text represents. A separator that should remain in a whole URL may need encoding inside one value.
Use URL Encode to transform short strings one at a time. These experiments inspect representation without visiting the URL; example.invalid is an illustrative destination. Correct-looking encoding proves neither that a resource exists nor that a link is safe.
Compare two encodings of one space
red blueComponent encoding produces red%20blue; form mode produces red+blue. They belong to different conventions rather than one always being wrong. Plus represents a space where form rules apply, not everywhere in every URL. Check the receiving parser's rules.
Choose the matching decoding convention too. Ordinary decoding leaves red+blue unchanged; treating form plus signs as spaces yields red blue. Before decoding repeatedly to fix an unexpected result, establish whether the input is a plain component or a form value.
The plus signs in C++ are literal data
C++ guideC%2B%2B%20guide
C%2B%2B+guideEven in form mode, literal plus signs become %2B, distinguishing them from space-derived plus signs. Form-decoding the unencoded C++ guide would turn its literal plus signs into spaces. Keep raw inputs separate from values that have already been encoded.
Whether & stays depends on its role
tea & cakeComponent encoding yields tea%20%26%20cake. Concatenating a raw & into a query may make it a parameter separator. Conversely, encoding every ? and & of a completed URL as one value destroys the intended structure. Distinguish constructing the URL from representing an individual value inside it.
Whole-URL mode preserves some structural punctuation; it does not infer intent and repair a malformed URL. Use URL Parser to inspect host, path, query, and fragment. In particular, a literal # in a query value must not accidentally begin a fragment.
%2520 can reveal a second encoding layer
red blue → red%20blue → red%2520blueThe second pass encodes the percent character as %25. One decoding pass then leaves red%20blue, which can look like decoding failed. The value alone cannot tell whether encoding happened twice by mistake or whether literal %20 was intended. Trace transformations on both sides.
Do not decode repeatedly until every percent sign disappears: intentional text such as %2F could become a separator and change meaning. The correct number of layers follows the data contract, not visual neatness. Distinguish the value before encoding, immediately before transmission, and after parsing.
Why one Japanese character becomes several % sequences
あ
UTF-8: e3 81 82
URL: %E3%81%82Each two-digit percent sequence represents one UTF-8 byte here. あ occupies three bytes and therefore three percent sequences; its Unicode code point U+3042 is not written as %3042. Text-to-bytes conversion helps connect the character, encoded bytes, and URL representation.
Truncated %E3%81 or nonhex %ZZ cannot decode correctly. Do not invent missing bytes just to remove an error; check for truncation at the source. If a legacy system uses another encoding, this decoder cannot automatically establish that original character set.
Define the encoding boundary in your implementation
Record the raw search term, selected convention, encoded value, and receiving convention. Passing red blue does not cover the bugs revealed by C++ guide or tea & cake. Keep small cases containing spaces, +, &, #, and % as regression examples when URL construction changes.
Structured URL/query APIs can reduce manual concatenation in code. But if the API encodes values, do not pre-encode them with the same transformation. The purpose of this experiment is not to add more conversions; it is to identify which layer owns the required conversion.
Separate an ampersand in a value from a query separator
For the search term red & blue, the ampersand belongs to the value. Appending it raw can make a receiver interpret it as another parameter boundary. Component encoding gives red%20%26%20blue; form encoding gives red+%26+blue. Both protect the literal ampersand as %26 while representing spaces differently. Preserving query separators and protecting characters inside values are separate responsibilities.
Compare the original value, assembled URL and received value. An address bar may display a friendlier representation, so appearance alone does not establish double encoding. Inspect query values with the URL parser and check for the original red & blue without visiting the address. Repeatedly decoding the whole URL can turn a protected %26 back into a structural &, so identify the layer before each conversion.
If only non-ASCII values fail, inspect which layer handles UTF-8 bytes before switching encodings. Percent pairs represent encoded bytes, not Unicode character numbers. A longer encoded string does not mean more original characters. Measure the original value and assembled URL separately when investigating length limits.