Using Binary SerializationSystem.Net.Mail How To

Beware of the Byte Order Mark

May 26th, 2007

Beware! This dog will byteVS 2005 includes several new file management wrappers in the My.Computer.FileSystem namespace, including WriteAllText. This method is a quick and handy way to write small sized strings to disk but this method has a ‘gotcha’ in it. It uses UTF-8 encoding with a Byte Order Mark by default. This mark can easily confuse programs that aren’t expecting it.

What is a Byte Order Mark? Simply put, it’s a set of characters at the beginning of a Unicode text file that denote how the file is encoded. For UTF-8, the characters are the byte sequence EF BB BF, which appears as the characters “”.

Many Windows programs and .NET functions handle this transparently for you so you never see these characters. For example, if you open a UTF-8 encoded file in Notepad, you won’t see these characters at all. This ‘helpfulness’ can make it difficult to spot a that the encoding of the file is a problem.

How can it be a problem? The problems come up when a program that expects ASCII encoding tries to read the file. For example, you might be sending files you create to a third party program. It loads the file and sees the Byte Order Mark and assumes that the file isn’t in the correct format and rejects it. If you load the file into Notepad it looks OK to you but the target program can’t read it. It can be a frustrating problem to track down.

WriteAllText is different from similar methods, System.IO.StreamWriter and System.IO.File.WriteAllText. They do write UTF-8 encoding by default but they do not include the Byte Order Mark. So, if you were using StreamWriter in VS 2002/2003 or used System.IO.File.WriteAllText elsewhere and switched to My.Computer.FileSystem.WriteAllText there is a difference in the output files you’re producing.

So, to avoid this kind of problem, use the overload for My.Computer.FileSystem.WriteAllText that includes encoding and avoid the one that doesn’t. In fact, my recommendation is to always specify the right encoding method no matter which method you use.

Use:

My.Computer.FileSystem.WriteAllText(MyFilename, MyString, False, System.Text.Encoding.ASCII)

System.IO.File.WriteAllText(MyFilename, MyString, False, System.Text.Encoding.ASCII)

Using sw As New StreamWriter(MyFilename, False, System.Text.Encoding.ASCII)
    'writing code here
End Using

OK (but not recommended):

System.IO.File.WriteAllText(MyFilename, MyString, False)

Using sw As New StreamWriter(MyFilename, False)
    'writing code here
End Using

Avoid:

My.Computer.FileSystem.WriteAllText(MyFilename, MyString, False)
Share This Article: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • del.icio.us
  • Reddit
  • StumbleUpon
  • Technorati
  • DotNetKicks
  • DZone

Entry Filed under: Tip Sheets


Rate This Article:

Not That GoodCould Be BetterOKGoodGreat (3 votes, average: 4.67 out of 5)
Loading ... Loading ...

4 Comments Add your own

  • 1. ASweet  |  October 26th, 2007 at 1:10 pm

    Very good, worked like a charm

  • 2. jfrankcarr  |  October 26th, 2007 at 1:42 pm

    You’re welcome. I’m glad to be of help.

  • 3. Alexey  |  November 29th, 2007 at 6:27 am

    System.IO.StreamWriter and System.IO.File.WriteAllText will not include the Byte Order Mark only if you use Encoding.Default parameter. It’s more safe to use custom Encoding:

    Dim encoding As New UTF8Encoding(False)

    Using sw As New StreamWriter(MyFilename, False, encoding)
    'writing code here
    End Using

  • 4. Mahdi aldurazi  |  October 25th, 2008 at 11:13 am

    Thanks,,

    its very helpfull information ..

Leave a Comment

Required

Required, hidden

Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Trackback this post  |  Subscribe to the comments via RSS Feed


Visit Me At My New Site, Programming In C#

Most Popular Articles

Highest Rated Articles

Categories

Most Recent Articles

Feeds

 Subscribe in a reader

To subscribe by e-mail
Enter your address here

Delivered by FeedBurner

VB Opportunities

Archives