Something went wrong. Try again.
[READ-ONLY] Mirror of https://github.com/VibeDevelopers/Comet.
Something went wrong. Try again.
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230BATCH support documentation========================================
Portions of BATCH are supported in comet core, whereas other portions areimplemented via the m_batch module. The portions in core are sufficient forserver-initiated batches sent to locally-connected clients. The moduleimplements client-initiated batches as well as batches that are propagatedbetween servers.
Contents:1. Specification2. Core support3. Server-initiated batches4. Client-initiated batches5. Propagated batches6. Other notes
========================================1. Specification========================================
Batches are specified as part of the IRCv3 effort. At the time of thiswriting, client-initiated batches are still in draft status.
- Core spec: https://ircv3.net/specs/extensions/batch- Client-initiated batches: https://ircv3.net/specs/extensions/client-batch- Netsplit/netjoin batch types: https://ircv3.net/specs/batches/netsplit
Individual batch types have their own specifications.
========================================2. Core support========================================
To support server-initiated batches, the "batch" client capability is defineddirectly in core via CLICAP_BATCH. If a client has this capability, they areable to receive arbitrary batch types. Portions of core use this to implementthe netsplit (in ircd/client.c) and netjoin (in modules/core/m_join.c) batchtypes.
The netjoin batch type additionally depends on support added to ircd/channel.cto send a JOIN message to a client with an included batch tag. This ensuresthat only JOINs that are actually part of a netjoin are tagged with the batch,and other normal JOIN messages lack the tag. The netjoin batch is triggered byreceiving an SJOIN command from a remote server that has not yet sent an EOB.
Some support for client-initiated and propagated batches are also in core.The LocalUser struct in includes/client.h received two additional fields tosupport queueing messages for future processing. A new hook named"message_handler" was added as well to override which handler is used whenparsing a command, to be used for the queueing support as well. Finally,functions for registering and unregistering recognized client-initiated andpropagated batch types along with their handler is present in core.
The two additional fields of LocalUser are `unsigned int pending_batch_lines`and `rb_dlink_list pending_batches`. pending_batch_lines keeps track of thenumber of messages currently queued for future processing. These messages arecounted against the client's receive queue, so a client with too many combinedlines between queued messages from incomplete batches and lines that simplyhaven't been parsed yet (perhaps due to fakelag) will be disconnected with"Excess Flood". In this way, we cap the overall memory that the ircd will beallocating for a client and ensure that batches cannot be used as a way tocheat flood protection controls. pending_batches keeps track of each batchthat the client initiated but has not yet completed. The struct used for thelist nodes is defined in includes/batch.h (explained in a couple paragraphs).
The message_handler hook is called each time a command is parsed, before it isdispatched to the module that handles that command. The hook can be used tooverride the handler with a different one, allowing a different module tohandle the command instead. This is used by m_batch for client-initiatedbatches and that usage is explained more in that section.
Finally, a new header includes/batch.h contains helpful methods for bothserver-initiated and client-initiated/propagated batches. For the former, amethod generates a randomized 15-character batch ID into a buffer for use withBATCH commands and the batch message tag on outgoing messages. For the latter,two functions are provided for modules to register/unregister recognized batchtypes along with the handlers that should be called when a batch of that typeis received from a client or remote server. Attempting to register a type thathas already been registered is an error. The header finally defines a structthat contains relevant metadata about an incoming batch as well as functionsto allocate and free that structure and deep copy messages to add to a batch.
========================================3. Server-initiated batches========================================
Beyond CLICAP_BATCH, no special support exists in the server for batchesinitiated by the server to be sent to local clients. It is expected that userscan generate their own batch reference tags (whether randomly or using somesort of static or data-dependent value) and make use of appropriate existingsendto_* functions to send the BATCH commands to relevant clients as well asthe messages within the batch. The vast majority of sendto_* functions supportvariants that take message tags with their parameters so a module canexplicitly add the batch tag to the outgoing message without needing to relyon the "outbound_msgbuf" hook.
Core support exists ONLY for server-initiated batches being sent to localclients; the m_batch module is required if BATCH commands need to be sentbetween servers.
========================================4. Client-initiated batches========================================
This section concerns batches that are initiated by locally-connected clients.Batches made by remote clients are described in the next section.
While not mandated by the current draft specification for client-initiatedbatches, the m_batch module supports having multiple interleaved batches inflight by a single client, nested batches, and the ability for clients to sendmessages not associated with a batch while a batch is in flight. Messages sentby a client with a valid batch tag are queued into that respective batch, andwill not be processed by the server until the client closes that batch.
When the client initiates a new batch, m_batch checks if any module hasregistered a handler for that batch type. If there is no registered handler,the BATCH command is rejected. Any messages the client sends tagged with thatbatch and the closing BATCH command (if any is sent) are rejected as well. Ifthe incoming batch has a registered handler, a deep copy of the BATCH commandis made and attached to the client's pending_batches along with other metadataneeded to track the batch (such as when the batch expires). The client'spending_batch_lines are incremented by 1 due to this new allocation. Deepcopies (that is, copies that include copying all pointed-to values) are madebecause the BATCH command needs to be long-lived beyond the current messagehandler invocation and the pointer lifetimes of message parameters do notextend beyond the current handler invocation.
When a message is received from a local client with a batch tag, and the tagreferences a valid batch, the "message_handler" hook function replaces thedefault handler for that command with a handler that appends a deep copy ofthat message to the relevant node in the client's pending_batches list as wellas incrementing their pending_batch_lines by 1. If the tag does not referencea valid batch, the "message_handler" hook function replaces the defaulthandler with m_ignore to discard the message. The specification requires thisbehavior when a batch has timed out, and the module does not keep track ofexpired batch reference tags, so any invalid reference tag receives this sametreatment.
When the closing BATCH message is received with a valid reference tag, them_batch module once again looks up the handler for that batch type. We do notsave the batch handler from the initial message due to the risk of the modulehandling that batch type being unloaded between the time the client begins abatch and ends that batch. In the event that the batch is no longer valid, themessages are dropped (as if they were originally received with a batch messagetag containing an invalid reference). Otherwise, the handler is called. Ineither case, the batch is removed from the client's pending_batches list andthe client's pending_batch_lines are reduced by the number of messagesassociated with the batch.
No fake lag is applied to processing the messages inside of a batch when theclient completes a batch, as fake lag was already applied to the clientadding the messages to the batch and the client's receive queue was stillcounting those messages against the total.
Client-initiated batches cannot remain open indefinitely. Approximately every30 seconds, a timer executes which forcibly closes batches that have been openfor longer than 15 seconds. This cleanup is non-essential as it is expectedmost clients will complete batches long before this timeout period. Because weneed to iterate over every local client and then every batch pending by thoseclients, the timer does not run with high frequency. This means that clientshave anywhere between 15-45 seconds (approximately) to finish a batch. If abatch is timed out, it is removed from the client's list of pending_batches,the number of lines in the batch are deducted from the client'spending_batch_lines, the messages in the batch are discarded without beingprocessed, and the client is given a notification that the batch timed out.
If a client is nesting batches and completes the outer batch before the nestedinner batch, the inner batch will be discarded without being processed and theclient will be given a notification that they did not properly close the innerbatch.
Empty client-initiated batches will still trigger relevant handlers. Handlersshould check batch->len to verify if the batch was empty or not; it will be 0for empty batches.
Exiting clients will cause all pending batches associated with that client tobe silently aborted, with no output being sent to the client (since they'regone anyway).
========================================5. Propagated batches========================================
The server protocol BATCH command is equivalent in syntax to the clientprotocol BATCH command. Like client batches, propagated batches are deferredfor processing until the batch is complete. Once complete, the batch handlerwill be called on the full batch if the batch type is recognized. If the batchtype is not recognized, it will send a FAIL message to the source if the sourceis a client, otherwise it will silently drop the message.
Batch types should only be sent to other servers if the sending server isassured that the batch type is available, such as via a server capability sentvia the CAPAB command for that batch type. The m_batch module does not defineany propagated batch types or server capabilities.
========================================6. Other notes========================================
If the m_batch module is unloaded, all pending client-initiated and propagatedbatches are aborted without their messages being processed. This will resultin the pending_batches list becoming empty and pending_batch_lines being 0 forall locally-connected clients and servers. Local clients will be notified thattheir batches have timed out.
Errors with client-initiated batches are sent to clients using thestandard-replies framework specified in IRCv3. The codes used by m_batch areas follows:
- FAIL BATCH TIMEOUT <reference-tag>: Indicates a client-initiated batch timed out or the m_batch module was unloaded.- FAIL BATCH INVALID_NESTING <reference-tag> <parent-type> <type>: Indicates that the specified batch type is not allowed to be nested under the parent's batch type (via the child_allowed callback or BATCH_FLAG_ALLOW_ALL).- FAIL BATCH INVALID_REFTAG <reference-tag>: Indicates the batch contains an invalid reference tag (either creating a new batch with an already-existing tag, not beginning the tag with '+' or '-', the tag consists solely of "+" or "-", or closing a batch with a tag not associated with an open batch).- FAIL BATCH UNKNOWN_TYPE <reference-tag> <type>: Indicates the batch type is not recognized by a module on the ircd.- FAIL BATCH INCOMPLETE <reference-tag>: Indicates the client is using nested batches and closed an outer batch before closing the inner nested batch.
Each batch is given a server-generated reference tag in addition to the tagspecified by the client or remote server. When the batch is propagated, thelocally-generated reference tag is used rather than the client-specified one.Doing this solves any moderation considerations regarding the contents of aclient-specified batch reference tag being shared with other clients.