Something went wrong. Try again.
Markdown parser fork with extended syntax for personal use.
Something went wrong. Try again.
1.6 kB · 67 lines
Rust
at main
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768//! Byte order mark occurs at the start of the document.//!//! ## Grammar//!//! Byte order mark forms with the following BNF//! (<small>see [construct][crate::construct] for character groups</small>)://!//! ```bnf//! byte_order_mark ::= 0xEF 0xBB 0xBF//! ```//!//! ## Recommendation//!//! Don’t use BOMs.//!//! ## Tokens//!//! * [`ByteOrderMark`][Name::ByteOrderMark]//!//! ## References//!//! * [`micromark/lib/preprocess.js` in `micromark`](https://github.com/micromark/micromark/blob/ed23453/packages/micromark/dev/lib/preprocess.js#L54-L60)
use crate::event::Name;use crate::state::{Name as StateName, State};use crate::tokenizer::Tokenizer;
/// Bytes of a BOM.const BOM: [u8; 3] = [0xEF, 0xBB, 0xBF];
/// Before BOM.////// ```text/// > | 0xEF 0xBB 0xBF/// ^^^^/// ```pub fn start(tokenizer: &mut Tokenizer) -> State { if tokenizer.current == Some(BOM[0]) { tokenizer.enter(Name::ByteOrderMark); State::Retry(StateName::BomInside) } else { State::Nok }}
/// In BOM.////// ```text/// > | 0xEF 0xBB 0xBF/// ^^^^ ^^^^ ^^^^/// ```pub fn inside(tokenizer: &mut Tokenizer) -> State { if tokenizer.current == Some(BOM[tokenizer.tokenize_state.size]) { tokenizer.tokenize_state.size += 1; tokenizer.consume(); if tokenizer.tokenize_state.size == BOM.len() { tokenizer.exit(Name::ByteOrderMark); tokenizer.tokenize_state.size = 0; State::Ok } else { State::Next(StateName::BomInside) } } else { tokenizer.tokenize_state.size = 0; State::Nok }}