Streaming network resources into GPU resources #6968

Activity

1. kainino0x

   added

   apiWebGPU APIWebGPU API

   on Aug 27, 2026
2. Kangz

   added this to the Milestone 3 milestone on Sep 23, 2026
3. shaoboyan091 commented on Oct 24, 2026

   

   shaoboyan091

   on Oct 24, 2026

   Last edited by shaoboyan091

   Contributor

   More actions

   LLM inference engine(e.g. ORT WebGPU EP) also hit same issues during model loading phase.

   Background

   Model are usually stored as files with different layout. Overall, these files contains two major part: Metadata to describe the model structure and Weights required by models. Weights are huge.

   Upload weights from file to GPU resource contains several copies. Take ORT WebGPU EP(Web) as an example, it's current uploading path on web version is:

   Blob wraps required file slice and create BYOB stream reader --> reader accumulate bytes into CPU memory (ArrayBuffer) --> Copy CPU memory to mapped staging GPUBuffer -> Use CopyBufferToBuffer to copy contents from staging buffer
   

   And ORT WebGPU EP(Native) has optimization on this workflow by providing mapped buffer space as read file destination to reduce 1 CPU-CPU copy. And the result shows 50%+ performance boost(NV: 1.562s -> 0.715s, Intel: 1.420 s -> 0.669 s) during model loading phase on test machines with Intel Core Ultra 9 285K + Intel Graphics (0x8086:0x7D67) + NVIDIA GeForce RTX 4070 (0x10DE:0x2709).

   Current constrains

   Using Stream API to read from file and copy to GPUBuffer is the current uploading path in ORT WebGPU EP(Web). But GPUBuffer mapping and Stream API has constraints to achieve similar approach in native because ReadableStreamBYOBReader transfers ownership of the destination’s underlying ArrayBuffer, while WebGPU forbids that transfer for mapped buffer.

   Proposals

   To achieve this goal we have several proposals

   Proposal A: Extend writeBuffer to accept Blob for reading from file cases

   Using Blob instead of ReadableStream to keep WriteBuffer sync semantic and handle the reading file to mapped buffer details in browser implementation. Blob could be created from blobs, which provide chances to wrap multiple file slices into one and let browser implementation to decide how to handle them, e.g. using multiple thread to do uploading in renderer process.

   Proposal B : New API GPUBuffer.mapWriteFromStreamAsync

   Reading from stream is async operation and is not feasible to have a queue command to handle it. So we proposed an API in GPUBuffer:

     partial interface GPUBuffer {
         Promise<undefined> mapWriteFromStreamAsync(
             ReadableStream source,
             optional GPUSize64 offset = 0,
             optional GPUSize64 size
         );
     };
   

   The API defaults the GPUMapModeFlags to WRITE. Reading from stream using mapped buffer as destination needs to be handled by browser implementation. In browser, we use BYOBStreamReader mechanism to read bytes into mapped buffer and keeps GPUBuffer map state as "pending".
   Overall, this API follows mapAsync validation rules. And details:

   • At the start: The buffer must be "unmapped", with the same usage, alignment, and range validation as mapAsync(WRITE, ...). An already mapped buffer or an existing pending mapping causes the new request to
     reject.

   • While streaming: mapState stays "pending". The browser acquires a reader for the source and writes incoming bytes into the requested range internally. As with mapAsync, it waits for previously submitted GPU
     work using the buffer before starting writable access. getMappedRange() remains unavailable during this period.

   • On success: Once the stream has completed successfully and its data has been written, the buffer transitions to "mapped" and the promise resolves with undefined. The application can then use
     getMappedRange(), or simply call unmap() before submitting GPU work using the buffer.

   • Concurrent operations: Another mapAsync() or mapWriteFromStreamAsync() call rejects with OperationError and generates a validation error, without interrupting the original operation. Submitting new GPU work
     using the buffer while it is pending or mapped fails validation, following the existing mapping rules.

   • Interruption: Calling unmap() (https://gpuweb.github.io/gpuweb/#dom-gpubuffer-unmap) while streaming is pending cancels the operation, rejects its promise with AbortError, and immediately sets mapState to
     "unmapped". The browser stops further writes, requests cancellation of the source stream through its reader, and releases the reader’s lock. Partial upload data is discarded. Calling destroy() also cancels streaming and
     destroys the buffer. After streaming completes successfully, ordinary unmap() preserves the uploaded data.

   • Stream failure: If reading the ReadableStream fails, the operation rejects its promise with the stream’s original failure reason, releases the reader, and returns the buffer to "unmapped". A stream failure
     itself would not generate a GPU validation error. Partially written data should be discard/cleared.
4. kainino0x commented on Oct 28, 2026

   

   kainino0x

   on Oct 28, 2026

   ContributorAuthor

   More actions

   writeBuffer is a synchronous/atomic operation so I don't think it makes sense to generalize it to receiving a stream - it would still need to put the buffer in some "pending" state to prevent other stuff from happening while the stream is downloading.

   Also I don't think it's possible to get a Blob until after the download completes but we would want to be able to stream into the resource while downloading.

   mapWriteFromStreamAsync probably makes sense but could you go into more detail about how it might work? Does the buffer need to be mapped or unmapped at the start? What is the state of the buffer after it finishes? What happens if you try to unmap, map, or use the buffer while it's streaming? Is there a way to interrupt it? What happens if the ReadableStream fails?
5. shaoboyan091 commented on Oct 29, 2026

   

   shaoboyan091

   on Oct 29, 2026

   Last edited by shaoboyan091

   Contributor

   More actions

   Also I don't think it's possible to get a Blob until after the download completes but we would want to be able to stream into the resource while downloading.

   Yes, and blob is feasible to the case that we already have a model in local or in cache but cold start the inference framework. That's the main use cases. And I don't think writeBuffer is feasible to stream too.

   you go into more detail about how it might work?

   Will add more details in the comments(added)