Dpnp Io

dpnp file I/O

Purpose

Gets data in and out of dpnp arrays. dpnp has no native binary file I/O: every format goes through NumPy, with dpnp.array() on the way in and dpnp.asnumpy() on the way out. This skill is that round trip, plus the chunked variants for data larger than memory and the format choice by size.

Prefer it over reaching for a dpnp.save() that does not exist, and over loading a file whole when the device cannot hold it.

When to Use This Skill

Use this skill when:

• An array has to be loaded into dpnp from a file, or a result written out.
• A file is larger than host or device memory and must be streamed in pieces.
• The user is choosing between .npy, .npz, HDF5, Zarr, and CSV.
• The user asks why dpnp will not write their format.

Do not use this skill to decide device placement or chunk sizing against device capacity — that is dpnp-memory — and do not expect it to make an I/O-bound job faster: if reading dominates, moving the compute to a device changes nothing.

Quick Start

import numpy
import dpnp

arr = dpnp.array(numpy.load("data.npy"))          # host file -> device array
result = dpnp.fft.fft2(arr) + dpnp.mean(arr)      # compute on the device
numpy.save("output.npy", dpnp.asnumpy(result))    # device array -> host file

The whole skill is that shape: NumPy load → dpnp.array() → compute → dpnp.asnumpy() → NumPy save.

Implementation Guide

1. .npy and .npz. One array or several, with the archive closed after reading:

   with numpy.load("data.npz") as npz:
       x = dpnp.array(npz["x"])
       y = dpnp.array(npz["y"])
   
   numpy.savez("output.npz", x=dpnp.asnumpy(x), y=dpnp.asnumpy(y))
   

   Each conversion needs a full host copy of the array as well as the device copy, so a 4 GB array wants 4 GB of free RAM during the call.

2. Chunked reads for a file larger than RAM. Memory-map the source, write each processed chunk straight into a pre-allocated output slice rather than appending to a list:

   data = numpy.load("large.npy", mmap_mode="r")
   final = numpy.empty(len(data), dtype=numpy.float64)
   chunk = 25_000_000
   
   for start in range(0, len(data), chunk):
       host = data[start:start + chunk]
       processed = dpnp.sqrt(dpnp.array(host)) * 2.0
       final[start:start + len(host)] = dpnp.asnumpy(processed)
   
   numpy.save("output.npy", final)
   

   A chunk of roughly a tenth to a fifth of free RAM is a workable start.

3. HDF5 through h5py. h5py only speaks NumPy, so the same conversion applies, and datasets can be written incrementally when the result is too large to hold:

   import h5py
   
   with h5py.File("output.h5", "w") as handle:
       dset = handle.create_dataset("result", shape=(50_000_000,), dtype="float64")
       for start in range(0, 50_000_000, 5_000_000):
           dset[start:start + 5_000_000] = dpnp.asnumpy(compute_chunk(start))
   

4. Zarr for very large or remote arrays. Chunked, compressed, and reachable on object storage through fsspec; read and write slice by slice:

   import zarr
   
   store = zarr.open("output.zarr", mode="w", shape=(10_000_000,),
                     chunks=(500_000,), dtype="float32")
   for start in range(0, 10_000_000, 500_000):
       store[start:start + 500_000] = dpnp.asnumpy(compute_chunk(start))
   

5. Text and CSV. dpnp.loadtxt() returns a dpnp array directly (it delegates to numpy.loadtxt internally, and does not support structured dtypes). Anything with headers, strings, or missing values goes through numpy.loadtxt/numpy.genfromtxt or pandas first:

   import pandas
   
   frame = pandas.read_csv("data.csv")
   arr = dpnp.array(frame.values)
   numpy.savetxt("output.csv", dpnp.asnumpy(arr), delimiter=",")
   

6. Pick the format by size. .npy/.npz below about a gigabyte, HDF5 for multi-dataset files in the gigabyte range, Zarr above that or when the data lives in cloud storage, CSV only for small human-readable exports.

Performance

No measured numbers ship with this skill. What to measure, and in which order:

• Time the I/O and the compute separately first. If reading dominates, no device will help and the conversion cost is irrelevant either way.
• Count conversions, not bytes. One conversion at each end of a batch of work is the pattern; one per iteration of a loop is the anti-pattern, and it is the usual reason a rewritten pipeline is no faster.
• Chunking trades peak memory against more conversions. Compare the two on the real file rather than assuming a ratio.
• CSV parsing is CPU-bound and dominates everything around it. Convert once to .npy or HDF5 if the same file is read repeatedly.

Gotchas & Limitations

• There is no dpnp.save() for binary formats. dpnp.loadtxt() exists; .npy, HDF5, and Zarr all go through NumPy. Code that calls a dpnp save function fails at the call, not at review.
• Conversion doubles peak memory. Host copy plus device copy, briefly, for every dpnp.array() and dpnp.asnumpy().
• Accumulating chunks in a list defeats chunking. The whole point is that the full array never exists in memory; a pre-allocated output or an incremental dataset write is what preserves that.
• mmap_mode="r" is a NumPy facility, not a device one. The mapped pages are host memory; each chunk still gets copied to the device.
• Not covered: parallel or multi-process writes, Arrow and Parquet, and anything about which device the array lands on — see dpnp-memory for that.

References

Two questions here should not be answered from memory: which I/O entry points the installed dpnp actually has (the list has grown between releases) and the current chunking API of h5py and Zarr, both of which are documented upstream and change on their own schedule.