← 输入框|github.com|刷新
Skip to content

An attempt to make asyncio.transport.writelines (selector) use Scatter I/O #84188

Closed

Description

@tzickel
mannequin
BPO 40007
Nosy @asvetlov, @1st1, @tzickel, @jakirkham
PRs
  • bpo-40007: Make asyncio.transport.writelines on selector use sendmsg #19062
  • Superseder
  • bpo-47010: Implement zero copy writes in SelectorSocketTransport in asyncio
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2022-03-17.13:11:02.294>
    created_at = <Date 2020-03-18.21:00:01.366>
    labels = ['type-feature', '3.9', 'expert-asyncio']
    title = 'An attempt to make asyncio.transport.writelines (selector) use Scatter I/O'
    updated_at = <Date 2022-03-17.13:11:02.294>
    user = 'https://github.com/tzickel'

    bugs.python.org fields:

    activity = <Date 2022-03-17.13:11:02.294>
    actor = 'asvetlov'
    assignee = 'none'
    closed = True
    closed_date = <Date 2022-03-17.13:11:02.294>
    closer = 'asvetlov'
    components = ['asyncio']
    creation = <Date 2020-03-18.21:00:01.366>
    creator = 'tzickel'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 40007
    keywords = []
    message_count = 3.0
    messages = ['364565', '364595', '415409']
    nosy_count = 4.0
    nosy_names = ['asvetlov', 'yselivanov', 'tzickel', 'jakirkham']
    pr_nums = ['19062']
    priority = 'normal'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '47010'
    type = 'enhancement'
    url = 'https://bugs.python.org/issue40007'
    versions = ['Python 3.9']

    Activity

    1. tzickel commented on Mar 18, 2020

      tzickelmannequin
      MannequinAuthor

      I have a code that tries to be smart and prepare data to be chunked efficiently before sending, so I was happy to read about:

      https://docs.python.org/3/library/asyncio-protocol.html#asyncio.WriteTransport.writelines

      Only to see that it simply does:

      self.write(b''.join(lines))

      So I've attempted to write an version that uses sendmsg (scatter I/O) instead (will be attached in PR).

      What I've learnt is:

      1. It's hard to benchmark (If someone has an good example on checking if it's worth it, feel free to add such).
      2. sendmsg has an OS limit on how many items can be done in one call. If the user does not call writer.drain() it might have too many items in the buffer, in that case I concat them (might be an expensive operation ? but that should not be th enormal case).
      3. socket.socket.sendmsg can accept any bytes like iterable, but os.writev can only accept sequences, is that a bug ?
      4. This is for the selector stream socket for now.
    2. tzickel commented on Mar 19, 2020

      tzickelmannequin
      MannequinAuthor

      BTW, if wanted a much more simpler PR can be made, where writelines simply calls sendmsg on the input if no buffer exists, and if not only then concats and acts like the current code base.

    3. asvetlov commented on Mar 17, 2022

      @asvetlov
      Contributor

      A duplicate of bpo-47010

    4. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions