[IdHTTP]No mapping for the Unicode charset

[Lena]

Hi.
In windows all works good but in android I got:
No mapping for the Unicode charset in the target multi-byte code page
How fix?
Thanks.

Code: Select all

void __fastcall TForm1::Image1Click(TObject *Sender)
{

 std::unique_ptr<TMemoryStream> WelcomeINI(new TMemoryStream());
 try
  {
   IdHTTP1->Get(L"http://welcome.um.la/welcome.ini", WelcomeINI.get());
  }
  catch(Exception &E)
      {
       ShowMessage(L"Error.\n" + E.Message);
       return;
      }

 WelcomeINI->Position = 0;
   String path = "";
   #ifdef __ANDROID__
   path = System::Ioutils::TPath::Combine(System::Ioutils::TPath::GetTempPath(), L"welcome.ini");
   #elif _Windows
   path = System::Ioutils::TPath::Combine(System::Ioutils::TPath::GetTempPath(), L"welcome.ini");
   #endif

 try
   {
   WelcomeINI->SaveToFile(path);
   }
   catch(Exception &E)
      {
       ShowMessage(L"Error.\n" + E.Message);
       return;
      }

 if(FileExists(path))
    {

     String SectionNAME = "";

     std::unique_ptr<TIniFile> FileINI(new TIniFile(path));

     std::unique_ptr<TStringList> AlliniSection(new TStringList());
     FileINI->ReadSections(AlliniSection.get());

     std::unique_ptr<TStringList> ValuesIniSection(new TStringList());

     TListBoxItem *ListBoxItem;
     TListBoxGroupHeader *ListBoxGroupHeader;
     ListBox1->BeginUpdate();
     for (int i = 0; i < AlliniSection->Count; i++)
       {
        SectionNAME = AlliniSection->Strings[i];
                  ListBoxGroupHeader = new TListBoxGroupHeader(ListBox1);
        ListBoxGroupHeader->Text = SectionNAME;
        ListBox1->AddObject(ListBoxGroupHeader);

        FileINI->ReadSectionValues(SectionNAME, ValuesIniSection.get());
        for (int j = 0; j < ValuesIniSection->Count; j++)
         {
         ListBoxItem = new TListBoxItem(ListBox1);
         ListBoxItem->Text = ValuesIniSection->KeyNames[j];
         ListBoxItem->StylesData["detail.visible"] = TValue::From<bool>(false);
         ListBoxItem->ItemData->Detail = ValuesIniSection->ValueFromIndex[j];

         //(aNone=0, aMore=1, aDetail=2, aCheckmark=3)
         ListBoxItem->ItemData->Accessory = static_cast<TListBoxItemData::TAccessory>(1);
         ListBox1->AddObject(ListBoxItem);
         }

       }
     ListBox1->EndUpdate();
    }
    else
       {
        ShowMessage(L"No file");
         }


}


If i save file welcome.ini in UTF-8 in android good but in Windows no. How fix code for android and windows?

[rlebeau]

In windows all works good but in android I got:
No mapping for the Unicode charset in the target multi-byte code page
How fix?

Which line of code is throwing that error? It can't be coming from TIdHTTP::Get(), because you are downloading the INI file to a TStream object, so TIdHTTP will not try to decode the response data as text at all.

Is the error coming from TIniFile? Since the INI file is encoded as UTF-8, I would suggest using TMemIniFile with TEncoding::UTF8 as the encoding:

Code: Select all

std::unique_ptr<TMemIniFile> FileINI(new TMemIniFile(path, TEncoding::UTF8));


Code: Select all

String path = "";
#ifdef __ANDROID__
path = System::Ioutils::TPath::Combine(System::Ioutils::TPath::GetTempPath(), L"welcome.ini");
#elif _Windows
path = System::Ioutils::TPath::Combine(System::Ioutils::TPath::GetTempPath(), L"welcome.ini");
#endif


You do realize how redundant those #ifdef statements are, don't you? You are using the exact same code for both Android and Windows, so you may as well not use the #ifdef's at all:

Code: Select all

String path = System::Ioutils::TPath::Combine(System::Ioutils::TPath::GetTempPath(), L"welcome.ini");


Code: Select all

if(FileExists(path))


That is also redundant, considering that you just saved the file you are about to load. Are you expecting someone to delete the file behind your back within a few milliseconds of saving the file? Besides, T(Mem)IniFile handles that check internally for you anyway, so you don't need to call it manually.

For that matter, do you really need to save the file to disk at all? Since the INI data is already in a TStream, you could alternatively load it into a TStringList and then load that into a TMemIniFile:

Code: Select all

std::unique_ptr<TMemIniFile> FileINI(new TMemIniFile(L""));
{
std::unique_ptr<TStringList> FileStrings(new TStringList);
WelcomeINI->Position = 0;
FileStrings->LoadFromStream(WelcomeINI.get(), TEncoding::UTF8);
FileINI->SetStrings(FileStrings.get());
}
...


If i save file welcome.ini in UTF-8 in android good but in Windows no.

You are contradicting yourself. Does it work on Windows and fail on Android, or the other way around?

On Window, TIniFile is a wrapper for Microsoft's PrivateProfile API. The Unicode version of that API works with UTF-8 data only if there is a UTF-8 BOM in the file (which your INI file does have).

On Android, TIniFile simply derives from TMemIniFile (to add a destructor that calls UpdateFile(), to match TIniFile on Windows). When TMemIniFile loads a file, it checks for a BOM when deciding which encoding to use (if not specified in the constructor) when reading/writing the file.

How fix code for android and windows?

Try using TMemIniFile instead of TIniFile. That way, you can ensure the INI data is decoded as UTF-8 on both platforms, and you can eliminate the file on disk completely (unless you need it for some other purpose). And also because the TIniFile destructor on both Windows and Android re-writes the INI file to save any changes you may have made to the INI data. Since you are not making any changes, the re-write is wasted overhead.

[Lena]

Thank you very much! I will try your recommendations.

You are contradicting yourself. Does it work on Windows and fail on Android, or the other way around?

1. Open notepad.exe and create ini file with russian text. Then save in ANSI (see picture).
Get this file with IdHTTP.
In Android application got:
No mapping for the Unicode charset in the target multi-byte code page
In Windows applicattion all good.

2. Open notepad.exe and create ini file with russian text. Then save in UTF- 8 (see picture).
Get this file with IdHTTP.
In Android applicattion all good.
In Windows application I do not see russian characters

Now I will try your recommendations with TMemIniFile...

[Lena]

For Android:
My new code works well if I save the file welcome.ini in notepad on the server encoding UTF-8.

Code: Select all

 void __fastcall TForm1::Image1Click(TObject *Sender)
{

 std::unique_ptr<TMemoryStream> WelcomeINI(new TMemoryStream());
 try
  {
   IdHTTP1->Get(L"http://welcome.um.la/welcome.ini", WelcomeINI.get());
  }
  catch(...)
      {
       ShowMessage(L"Check the Internet connection");
       return;
      }

  WelcomeINI->Position = 0;

  std::unique_ptr<TMemIniFile> FileINI(new TMemIniFile(L""));
  std::unique_ptr<TStringList> FileStrings(new TStringList);
  FileStrings->LoadFromStream(WelcomeINI.get(), TEncoding::UTF8);
  FileINI->SetStrings(FileStrings.get());

     String SectionNAME = "";
     std::unique_ptr<TStringList> AlliniSection(new TStringList());
     FileINI->ReadSections(AlliniSection.get());

     std::unique_ptr<TStringList> ValuesIniSection(new TStringList());

     TListBoxItem *ListBoxItem;
     TListBoxGroupHeader *ListBoxGroupHeader;

     ListBox1->BeginUpdate();
     for (int i = 0; i < AlliniSection->Count; i++)
       {
        SectionNAME = AlliniSection->Strings[i];
        ListBoxGroupHeader = new TListBoxGroupHeader(ListBox1);
        ListBoxGroupHeader->Text = SectionNAME;
        ListBox1->AddObject(ListBoxGroupHeader);
        Application->ProcessMessages();
        FileINI->ReadSectionValues(SectionNAME, ValuesIniSection.get());
        for (int j = 0; j < ValuesIniSection->Count; j++)
         {
         ListBoxItem = new TListBoxItem(ListBox1);
         //ListBoxItem->Height = 45;
         ListBoxItem->Text = ValuesIniSection->KeyNames[j];
         ListBoxItem->StylesData["detail.visible"] = TValue::From<bool>(false);
         ListBoxItem->ItemData->Detail = ValuesIniSection->ValueFromIndex[j];

         //(aNone=0, aMore=1, aDetail=2, aCheckmark=3)
         ListBoxItem->ItemData->Accessory = static_cast<TListBoxItemData::TAccessory>(1);
         ListBox1->AddObject(ListBoxItem);
         Application->ProcessMessages();
         }

       }
     ListBox1->EndUpdate();

}


But if someone forgets and saves the file welcome.ini encoding ANSI to get the android:
No mapping for the Unicode charset...

[rlebeau]

1. Open notepad.exe and create ini file with russian text. Then save in ANSI (see picture).
Get this file with IdHTTP.
In Android application got:
No mapping for the Unicode charset in the target multi-byte code page
In Windows applicattion all good.

On Android, TIniFile is an alias for TMemIniFile, which uses TEncoding for encoding/decoding text (TIniFile on Windows does not). Since an ANSI file has no BOM, and you are not specifying any TEncoding when creating the TIniFile object, it ends up using TEncoding.Default. On Android, that is UTF-8. On Windows, that is ANSI (which usually means the file will only work if loaded on a system that is using the same locale as the system that created it).

2. Open notepad.exe and create ini file with russian text. Then save in UTF- 8 (see picture).
Get this file with IdHTTP.
In Android applicattion all good.
In Windows application I do not see russian characters

It works on Android because TMemIniFile will end up expecting UTF-8 by default. But even if it weren't, Notepad saves UTF-8 text with a BOM, which would force TMemIniFile to use UTF-8 anyway.

It *should* be working on Windows, due to the UTF-8 BOM, which is suppose to be supported by the Win32 API PrivateProfile API (which TIniFile uses on Windows), although this is not really documented behavior. Make sure the BOM is actually present in the file.

For Android:
My new code works well if I save the file welcome.ini in notepad on the server encoding UTF-8.
...
But if someone forgets and saves the file welcome.ini encoding ANSI to get the android:
No mapping for the Unicode charset...

As it should be, per what I said above. If the file is not UTF-8, loading it as UTF-8 will fail. If the file is UTF-8, you must load it as UTF-8. If the file is ANSI, you must load it as ANSI (which is a bit more difficult to do when you cross over platform boundaries - ANSI on one system may be different than ANSI on another system). You can't really have it both ways, unless you analyze the raw bytes and *guess* what encoding is being used (which is easier to do if a BOM is present, but is very difficult to do without a BOM).

It is the user's responsibility to save the file in the correct encoding, and it is the server's responsibility to report the correct encoding in the HTTP "Content-Type" header (which the TStream version of TIdHTTP::Get() does not use, but you could if needed, provided it is accurate, which is probably not going to always be the case).

[Lena]

Thank you very much for the detailed information and help!