Bun Binary Data

Binary Data

Working with binary data in JavaScript

This page is intended as an introduction to working with binary data in JavaScript. Bun implements a number of data types and utilities for working with binary data, most of which are Web-standard. Any Bun-specific APIs will be noted as such.

Below is a quick "cheat sheet" that doubles as a table of contents. Click an item in the left column to jump to that section.

ArrayBuffer and views

Until 2009, there was no language-native way to store and manipulate binary data in JavaScript. ECMAScript v5 introduced a range of new mechanisms for this. The most fundamental building block is ArrayBuffer, a simple data structure that represents a sequence of bytes in memory.

// this buffer can store 8 bytes
const buf = new ArrayBuffer(8);

Despite the name, it isn't an array and supports none of the array methods and operators one might expect. In fact, there is no way to directly read or write values from an ArrayBuffer. There's very little you can do with one except check its size and create "slices" from it.

const buf = new ArrayBuffer(8);
buf.byteLength; // => 8

const slice = buf.slice(0, 4); // returns new ArrayBuffer
slice.byteLength; // => 4

To do anything interesting we need a construct known as a "view". A view is a class that wraps an ArrayBuffer instance and lets you read and manipulate the underlying data. There are two types of views: typed arrays and DataView.

DataView

The DataView class is a lower-level interface for reading and manipulating the data in an ArrayBuffer.

Below we create a new DataView and set the first byte to 3.

const buf = new ArrayBuffer(4);
// [0b00000000, 0b00000000, 0b00000000, 0b00000000]

const dv = new DataView(buf);
dv.setUint8(0, 3); // write value 3 at byte offset 0
dv.getUint8(0); // => 3
// [0b00000011, 0b00000000, 0b00000000, 0b00000000]

Now let's write a Uint16 at byte offset 1. This requires two bytes. We're using the value 513, which is 2 * 256 + 1; in bytes, that's 00000010 00000001.

dv.setUint16(1, 513);
// [0b00000011, 0b00000010, 0b00000001, 0b00000000]

console.log(dv.getUint16(1)); // => 513

We've now assigned a value to the first three bytes in our underlying ArrayBuffer. Even though the second and third bytes were created using setUint16(), we can still read each of its component bytes using getUint8().

console.log(dv.getUint8(1)); // => 2
console.log(dv.getUint8(2)); // => 1

Attempting to write a value that requires more space than is available in the underlying ArrayBuffer will cause an error. Below we attempt to write a Float64 (which requires 8 bytes) at byte offset 0, but there are only four total bytes in the buffer.

dv.setFloat64(0, 3.1415);
// ^ RangeError: Out of bounds access

The following methods are available on DataView:

TypedArray

Typed arrays are a family of classes that provide an Array-like interface for interacting with data in an ArrayBuffer. Whereas a DataView lets you write numbers of varying size at a particular offset, a TypedArray interprets the underlying bytes as an array of numbers, each of a fixed size.

const buffer = new ArrayBuffer(3);
const arr = new Uint8Array(buffer);

// contents are initialized to zero
console.log(arr); // Uint8Array(3) [0, 0, 0]

// assign values like an array
arr[0] = 0;
arr[1] = 10;
arr[2] = 255;
arr[3] = 255; // no-op, out of bounds

While an ArrayBuffer is a generic sequence of bytes, these typed array classes interpret the bytes as an array of numbers of a given byte size. The top row contains the raw bytes, and the later rows contain how these bytes will be interpreted when viewed using different typed array classes.

The following classes are typed arrays, along with a description of how they interpret the bytes in an ArrayBuffer:

Here's the first table formatted as a markdown table:

The table below demonstrates how the bytes in an ArrayBuffer are interpreted when viewed using different typed array classes.

To create a typed array from a pre-defined ArrayBuffer:

// create typed array from ArrayBuffer
const buf = new ArrayBuffer(10);
const arr = new Uint8Array(buf);

arr[0] = 30;
arr[1] = 60;

// all elements are initialized to zero
console.log(arr); // => Uint8Array(10) [ 30, 60, 0, 0, 0, 0, 0, 0, 0, 0 ];

If we tried to instantiate a Uint32Array from this same ArrayBuffer, we'd get an error.

const buf = new ArrayBuffer(10);
const arr = new Uint32Array(buf);
//          ^  RangeError: ArrayBuffer length minus the byteOffset
//             is not a multiple of the element size

A Uint32 value requires four bytes (16 bits). Because the ArrayBuffer is 10 bytes long, there's no way to cleanly divide its contents into 4-byte chunks.

To fix this, we can create a typed array over a particular "slice" of an ArrayBuffer. The Uint16Array below only "views" the first 8 bytes of the underlying ArrayBuffer. To achieve these, we specify a byteOffset of 0 and a length of 2, which indicates the number of Uint32 numbers we want our array to hold.

// create typed array from ArrayBuffer slice
const buf = new ArrayBuffer(10);
const arr = new Uint32Array(buf, 0, 2);

/*
  buf    _ _ _ _ _ _ _ _ _ _    10 bytes
  arr   [_______,_______]       2 4-byte elements
*/

arr.byteOffset; // 0
arr.length; // 2

You don't need to explicitly create an ArrayBuffer instance; you can instead directly specify a length in the typed array constructor:

const arr2 = new Uint8Array(5);

// all elements are initialized to zero
// => Uint8Array(5) [0, 0, 0, 0, 0]

Typed arrays can also be instantiated directly from an array of numbers, or another typed array:

// from an array of numbers
const arr1 = new Uint8Array([0, 1, 2, 3, 4, 5, 6, 7]);
arr1[0]; // => 0;
arr1[7]; // => 7;

// from another typed array
const arr2 = new Uint8Array(arr);

Broadly speaking, typed arrays provide the same methods as regular arrays, with a few exceptions. For example, push and pop are not available on typed arrays, because they would require resizing the underlying ArrayBuffer.

const arr = new Uint8Array([0, 1, 2, 3, 4, 5, 6, 7]);

// supports common array methods
arr.filter(n => n > 128); // Uint8Array(1) [255]
arr.map(n => n * 2); // Uint8Array(8) [0, 2, 4, 6, 8, 10, 12, 14]
arr.reduce((acc, n) => acc + n, 0); // 28
arr.forEach(n => console.log(n)); // 0 1 2 3 4 5 6 7
arr.every(n => n < 10); // true
arr.find(n => n > 5); // 6
arr.includes(5); // true
arr.indexOf(5); // 5

Refer to the MDN documentation for more information on the properties and methods of typed arrays.

Uint8Array

It's worth specifically highlighting Uint8Array, as it represents a classic "byte array"—a sequence of 8-bit unsigned integers between 0 and 255. This is the most common typed array you'll encounter in JavaScript.

In Bun, and someday in other JavaScript engines, it has methods available for converting between byte arrays and serialized representations of those arrays as base64 or hex strings.

new Uint8Array([1, 2, 3, 4, 5]).toBase64(); // "AQIDBA=="
Uint8Array.fromBase64("AQIDBA=="); // Uint8Array(4) [1, 2, 3, 4, 5]

new Uint8Array([255, 254, 253, 252, 251]).toHex(); // "fffefdfcfb=="
Uint8Array.fromHex("fffefdfcfb"); // Uint8Array(5) [255, 254, 253, 252, 251]

It is the return value of TextEncoder#encode, and the input type of TextDecoder#decode, two utility classes designed to translate strings and various binary encodings, most notably "utf-8".

const encoder = new TextEncoder();
const bytes = encoder.encode("hello world");
// => Uint8Array(11) [ 104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100 ]

const decoder = new TextDecoder();
const text = decoder.decode(bytes);
// => hello world

Buffer

Bun implements Buffer, a Node.js API for working with binary data that pre-dates the introduction of typed arrays in the JavaScript spec. It has since been re-implemented as a subclass of Uint8Array. It provides a wide range of methods, including several Array-like and DataView-like methods.

const buf = Buffer.from("hello world");
// => Buffer(11) [ 104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100 ]

buf.length; // => 11
buf[0]; // => 104, ascii for 'h'
buf.writeUInt8(72, 0); // => ascii for 'H'

console.log(buf.toString());
// => Hello world

For complete documentation, refer to the Node.js documentation.

Blob

Blob is a Web API commonly used for representing files. Blob was initially implemented in browsers (unlike ArrayBuffer which is part of JavaScript itself), but it is now supported in Node and Bun.

It isn't common to directly create Blob instances. More often, you'll receive instances of Blob from an external source (like an <input type="file"> element in the browser) or library. That said, it is possible to create a Blob from one or more string or binary "blob parts".

const blob = new Blob(["<html>Hello</html>"], {
  type: "text/html",
});

blob.type; // => text/html
blob.size; // => 19

These parts can be string, ArrayBuffer, TypedArray, DataView, or other Blob instances. The blob parts are concatenated together in the order they are provided.

const blob = new Blob([
  "<html>",
  new Blob(["<body>"]),
  new Uint8Array([104, 101, 108, 108, 111]), // "hello" in binary
  "</body></html>",
]);

The contents of a Blob can be asynchronously read in various formats.

await blob.text(); // => <html><body>hello</body></html>
await blob.bytes(); // => Uint8Array (copies contents)
await blob.arrayBuffer(); // => ArrayBuffer (copies contents)
await blob.stream(); // => ReadableStream

BunFile

BunFile is a subclass of Blob used to represent a lazily-loaded file on disk. Like File, it adds a name and lastModified property. Unlike File, it does not require the file to be loaded into memory.

const file = Bun.file("index.txt");
// => BunFile

File

Browser only. Experimental support in Node.js 20.

File is a subclass of Blob that adds a name and lastModified property. It's commonly used in the browser to represent files uploaded via a <input type="file"> element. Node.js and Bun implement File.

// on browser!
// <input type="file" id="file" />

const files = document.getElementById("file").files;
// => File[]

const file = new File(["<html>Hello</html>"], "index.html", {
  type: "text/html",
});

Refer to the MDN documentation for complete docs information.

Streams

Streams are an important abstraction for working with binary data without loading it all into memory at once. They are commonly used for reading and writing files, sending and receiving network requests, and processing large amounts of data.

Bun implements the Web APIs ReadableStream and WritableStream.

To create a simple readable stream:

const stream = new ReadableStream({
  start(controller) {
    controller.enqueue("hello");
    controller.enqueue("world");
    controller.close();
  },
});

The contents of this stream can be read chunk-by-chunk with for await syntax.

for await (const chunk of stream) {
  console.log(chunk);
}

// => "hello"
// => "world"

For a more complete discussion of streams in Bun, see API > Streams.

Conversion

Converting from one binary format to another is a common task. This section is intended as a reference.

From ArrayBuffer

Since ArrayBuffer stores the data that underlies other binary structures like TypedArray, the snippets below are not converting from ArrayBuffer to another format. Instead, they are creating a new instance using the data stored underlying data.

To TypedArray

new Uint8Array(buf);

To DataView

new DataView(buf);

To Buffer

// create Buffer over entire ArrayBuffer
Buffer.from(buf);

// create Buffer over a slice of the ArrayBuffer
Buffer.from(buf, 0, 10);

To string

As UTF-8:

new TextDecoder().decode(buf);

To number[]

Array.from(new Uint8Array(buf));

To Blob

new Blob([buf], { type: "text/plain" });

To ReadableStream

The following snippet creates a ReadableStream and enqueues the entire ArrayBuffer as a single chunk.

new ReadableStream({
  start(controller) {
    controller.enqueue(buf);
    controller.close();
  },
});

const view = new Uint8Array(buf);
const chunkSize = 1024;

new ReadableStream({
  start(controller) {
    for (let i = 0; i < view.length; i += chunkSize) {
      controller.enqueue(view.slice(i, i + chunkSize));
    }
    controller.close();
  },
});

From TypedArray

To ArrayBuffer

This retrieves the underlying ArrayBuffer. Note that a TypedArray can be a view of a slice of the underlying buffer, so the sizes may differ.

arr.buffer;

To DataView

To creates a DataView over the same byte range as the TypedArray.

new DataView(arr.buffer, arr.byteOffset, arr.byteLength);

To Buffer

Buffer.from(arr);

To string

As UTF-8:

new TextDecoder().decode(arr);

To number[]

Array.from(arr);

To Blob

// only if arr is a view of its entire backing TypedArray
new Blob([arr.buffer], { type: "text/plain" });

To ReadableStream

new ReadableStream({
  start(controller) {
    controller.enqueue(arr);
    controller.close();
  },
});

new ReadableStream({
  start(controller) {
    for (let i = 0; i < arr.length; i += chunkSize) {
      controller.enqueue(arr.slice(i, i + chunkSize));
    }
    controller.close();
  },
});

From DataView

To ArrayBuffer

view.buffer;

To TypedArray

Only works if the byteLength of the DataView is a multiple of the BYTES_PER_ELEMENT of the TypedArray subclass.

new Uint8Array(view.buffer, view.byteOffset, view.byteLength);
new Uint16Array(view.buffer, view.byteOffset, view.byteLength / 2);
new Uint32Array(view.buffer, view.byteOffset, view.byteLength / 4);
// etc...

To Buffer

Buffer.from(view.buffer, view.byteOffset, view.byteLength);

To string

As UTF-8:

new TextDecoder().decode(view);

To number[]

Array.from(view);

To Blob

new Blob([view.buffer], { type: "text/plain" });

To ReadableStream

new ReadableStream({
  start(controller) {
    controller.enqueue(view.buffer);
    controller.close();
  },
});

new ReadableStream({
  start(controller) {
    for (let i = 0; i < view.byteLength; i += chunkSize) {
      controller.enqueue(view.buffer.slice(i, i + chunkSize));
    }
    controller.close();
  },
});

From Buffer

To ArrayBuffer

buf.buffer;

To TypedArray

new Uint8Array(buf);

To DataView

new DataView(buf.buffer, buf.byteOffset, buf.byteLength);

To string

As UTF-8:

buf.toString();

As base64:

buf.toString("base64");

As hex:

buf.toString("hex");

To number[]

Array.from(buf);

To Blob

new Blob([buf], { type: "text/plain" });

To ReadableStream

new ReadableStream({
  start(controller) {
    controller.enqueue(buf);
    controller.close();
  },
});

new ReadableStream({
  start(controller) {
    for (let i = 0; i < buf.length; i += chunkSize) {
      controller.enqueue(buf.slice(i, i + chunkSize));
    }
    controller.close();
  },
});

From Blob

To ArrayBuffer

The Blob class provides a convenience method for this purpose.

await blob.arrayBuffer();

To TypedArray

await blob.bytes();

To DataView

new DataView(await blob.arrayBuffer());

To Buffer

Buffer.from(await blob.arrayBuffer());

To string

As UTF-8:

await blob.text();

To number[]

Array.from(await blob.bytes());

To ReadableStream

blob.stream();

From ReadableStream

It's common to use Response as a convenient intermediate representation to make it easier to convert ReadableStream to other formats.

stream; // ReadableStream

const buffer = new Response(stream).arrayBuffer();

However this approach is verbose and adds overhead that slows down overall performance unnecessarily. Bun implements a set of optimized convenience functions for converting ReadableStream various binary formats.

To ArrayBuffer

// with Response
new Response(stream).arrayBuffer();

// with Bun function
Bun.readableStreamToArrayBuffer(stream);

To Uint8Array

// with Response
new Response(stream).bytes();

// with Bun function
Bun.readableStreamToBytes(stream);

To TypedArray

// with Response
const buf = await new Response(stream).arrayBuffer();
new Int8Array(buf);

// with Bun function
new Int8Array(Bun.readableStreamToArrayBuffer(stream));

To DataView

// with Response
const buf = await new Response(stream).arrayBuffer();
new DataView(buf);

// with Bun function
new DataView(Bun.readableStreamToArrayBuffer(stream));

To Buffer

// with Response
const buf = await new Response(stream).arrayBuffer();
Buffer.from(buf);

// with Bun function
Buffer.from(Bun.readableStreamToArrayBuffer(stream));

To string

As UTF-8:

// with Response
await new Response(stream).text();

// with Bun function
await Bun.readableStreamToText(stream);

To number[]

// with Response
const arr = await new Response(stream).bytes();
Array.from(arr);

// with Bun function
Array.from(new Uint8Array(Bun.readableStreamToArrayBuffer(stream)));

Bun provides a utility for resolving a ReadableStream to an array of its chunks. Each chunk may be a string, typed array, or ArrayBuffer.

// with Bun function
Bun.readableStreamToArray(stream);

To Blob

new Response(stream).blob();

To ReadableStream

To split a ReadableStream into two streams that can be consumed independently:

const [a, b] = stream.tee();