If you:
- Have no idea what a stream is
- Have a blurry idea but need a better analogy
- Don't know how to use(understand) streams in actual code
This post is for you.
This is my brain trying to understand streams last week.
What Are Streams?
A Stream is the flow of data(bytes) that moves from one point to another, just like a stream of water flowing down a river.
Streams are used to read and write data to and from:
- Files
- Networks
- Memory
- Other streams
An Analogy To Help Us Navigate The Examples
Imagine a huge tank of muddy water(you don't know how much water is in there) and you need to clean this water and fill another tank with clean water.
This tank has a faucet on the outside that you can open and let the muddy water flow out.
You find a hose that has a water filter in it.
You connect the hose to the muddy water tank and put the other end in the clean water tank.
As you turn on the tap, the water flows through the hose, is filtered and clean water comes out.
- The stream is the flow of water(not the water itself).
- The muddy tank is the source you are trying to read(file in our case).
- The filter is a helper class that helps us clean the water(convert bytes to text).
- The clean water tank is like the output on the other end(in our case we'll be logging the file content to the console as it comes)
C# classes to work with streams
These are the classes to use when reading and writing to and from different sources like files, memory, network, other streams, etc(I don't know what else).
Stream is the abstract class that provides methods to transfer bytes to and from the source(file, memory, network, etc). It is the core class to transfer bytes.
If a class wants to read or write from a source(file, memory, or network) it needs to implement the Stream class.
We'll only focus on the FileStream and the helper classes StreamWriter and StreamReader in this post.
How To Read a File Using the FileStream
Let's create a simple method that takes a string path and reads the content of the file and outputs it to the console
using System.IO;
public static void ReadFile(string path) //E.g. @"d:\text.txt"
{
var fileStream = new FileStream(path, FileMode.Open, FileAccess.Read);
var buffer = new byte[fileStream .Length];
fileStream.Read(buffer);
var result = Encoding.UTF8.GetString(buffer);
Console.WriteLine(result);
fileStream.Flush(); // ensure that any data still in the buffer(bytesArray) is written to the file
fileStream.Close(); //release any system resources associated with the object
}
- The file path is the muddy water in the tank(but instead of mud we have bytes)
- The FileStream object is the muddy water tank's faucet(you open and bytes flow out instead of muddy water)
- The Encoding.UTF8.GetString() method is the filter cleaning the muddy water. In this case, it converts bytes into text.
A simplified way to write this method is with the using block. When the variable you declared inside of the using falls out of scope, it calls the .Dispose() method which internally calls .Flush() and .Close() automatically for us.
using System.IO;
public static void ReadFile(string path) //E.g. @"d:\text.txt"
{
using (var fileStream = new FileStream(path, FileMode.Open, FileAccess.Read))
{
var bytesArray = new byte[fileStream.Length];
fileStream.Read(bytesArray);
var result = Encoding.UTF8.GetString(bytesArray);
Console.WriteLine(result);
}
}
What if I don't want to deal with bytes directly is there a way to make this more high-level?
Here is where the helper class StreamReader comes in.
Have a look at the same code as above but using StreamReader instead:
using System.IO;
public static void ReadFile(string path) //E.g. @"d:\text.txt"
{
using (var streamReader = new StreamReader(path))
{
var result = streamReader.ReadToEnd();
Console.WriteLine(result);
}
}
Note: This is only one way of doing it. There are others, e.g. File.ReadAllText(path).
In the above example, we don't even need to use the FileStream. The StreamReader deals with the FileStream under the hood to make our lives simpler.
Let's have a look at another example.
What if we are reading a file that is bigger than our available RAM?
One thing is for sure, we cannot read the whole file in memory at once! We have to use the Hannibal Lecter approach and do it piece by piece(forgive my analogy)
Here is one way you can do it(Lower level version):
using System.IO;
public static string ReadLargeFile(string path)
{
const int bufferSize = 4096; // read the file in 4KB chunks
var builder = new StringBuilder();
using (var fileStream = new FileStream(path, FileMode.Open, FileAccess.Read))
using (var streamReader = new StreamReader(fileStream))
{
char[] buffer = new char[bufferSize];
int bytesRead;
while ((bytesRead = streamReader.ReadBlock(buffer, 0, bufferSize)) > 0)
{
builder.Append(buffer, 0, bytesRead);
}
}
return builder.ToString();
}
Note that in the above example, we use one of the StreamReader constructors that take a FileStream as a parameter. We do that because we need more control over the stream in this case.
Can we do away with the ugly lower-level code?
Yes, here we go:
using System.IO;
public static string ReadLargeFile(string path)
{
StringBuilder sb = new StringBuilder();
using (FileStream fileStream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read))
using (StreamReader streamReader = new StreamReader(fileStream))
{
string line;
while ((line = streamReader.ReadLine()) != null)
{
sb.AppendLine(line);
}
}
return sb.ToString();
}
In the above case:
- We are not dictating the size of the buffer to be read at a time(just do it line by line, however much that is)
- we don't have to keep track of the bytes read.
It is much simpler to use.
What about writing a file? Let's have a look next.
How To Write A File Using The FileStream
This simple method takes a string path, creates a file a writes some content to it.
Using the FileStream
using System.IO;
public static void WriteText(string path) //E.g. @"d:\text.txt"
{
string text = "hey my dude!";
byte[] bytesArray = Encoding.UTF8.GetBytes(text);
using (var fileStream = new FileStream(path, FileMode.Create))
{
fileStream.Write(bytesArray);
}
}
Using the StreamWriter helper:
using System.IO;
public static void WriteText(string path) //E.g. @"d:\text.txt"
{
using (var s = new StreamWriter(path))
{
s.WriteLine(text);
}
}
Note: The File class provides some handy methods as well. The above example could've been written as: File.WriteAllText(path, text);
Let's raise the same size problem again.
What if you want to write a big ass file that is too large to fit in your RAM at once?
You can use the same break it up as you write the same way we did when reading:
Using the FileReader only:
public string WriteFile(string path, string content)
{
// Create a new file at the specified path
using (FileStream stream = new FileStream(path, FileMode.Create, FileAccess.Write))
{
// Write the content to the file in chunks
byte[] buffer = Encoding.UTF8.GetBytes(content);
int chunkSize = 1024;
for (int i = 0; i < buffer.Length; i += chunkSize)
{
int remainingBytes = buffer.Length - i;
int bytesToWrite = remainingBytes < chunkSize ? remainingBytes : chunkSize;
stream.Write(buffer, i, bytesToWrite);
}
}
}
Using FileReader and StreamWriter:
public static void WriteLargeFile(string path, string content)
{
int bufferSize = 4096;
byte[] buffer = new byte[bufferSize];
int bytesRead = 0;
using (var stream = new FileStream(path, FileMode.Create, FileAccess.Write, FileShare.None, bufferSize, FileOptions.SequentialScan))
{
using (var writer = new StreamWriter(stream))
{
int offset = 0;
while (offset < content.Length)
{
int bytesToWrite = Math.Min(bufferSize, content.Length - offset);
var chunk = content.Substring(offset, bytesToWrite);
writer.Write(chunk);
writer.Flush();
offset += bytesToWrite;
}
}
}
}
In this case, you can see that you still have to deal with the file chunks more directly.
Conclusion
I hope streams make more sense to you now as it does for me.
Thanks again for reading!
If you like this article:
Leave a comment (You can just say hi!)
Let's connect on Twitter @theguspear
Catch you next week,
Gus.
Top comments (0)