Template of a custom feed generator service for the Bluesky network in Ruby

disable message queueing in "save matching" mode

If only matching posts are saved, there will not be 100s of messages to save per second, they could be as rare as a few messages per day. So not only we don't want to batch them by 100, but save them immediately, but also we can't use the batch save point to save the cursor periodically either. (We'll use a separate counter instead of dividing msg.seq by 100, because with Jetstream the seq doesn't increase by 1 anymore.)

Changed files
+16 -6
app
+16 -6
app/firehose_stream.rb
··· 57 57 @sky.on_connecting { |u| log "Connecting to #{u}..." } 58 58 59 59 @sky.on_connect { 60 + @message_counter = 0 60 61 @replaying = !!(cursor) 61 62 log "Connected ✓" 62 63 } ··· 122 123 if @replaying 123 124 log "Replaying events since #{msg.time.getlocal} -->" 124 125 @replaying = false 126 + end 127 + 128 + @message_counter += 1 129 + 130 + if @message_counter % 100 == 0 131 + # save current cursor every 100 events 132 + save_cursor(msg.seq) 125 133 end 126 134 127 135 msg.operations.each do |op| ··· 224 232 puts text 225 233 end 226 234 227 - if @save_posts == :all || @save_posts && matched 235 + if @save_posts == :all 236 + # wait until we have 100 posts and then save them all in one insert, if possible 228 237 @post_queue << post 229 - end 230 238 231 - # wait until we have 100 posts and then save them all in one insert, if possible 232 - if @post_queue.length >= POSTS_BATCH_SIZE 233 - save_queued_posts 234 - save_cursor(@sky.cursor) 239 + if @post_queue.length >= POSTS_BATCH_SIZE 240 + save_queued_posts 241 + end 242 + elsif @save_posts == :matching && matched 243 + # save immediately because matched posts might be rare; we've already checked validations 244 + post.save!(validate: false) 235 245 end 236 246 237 247 print '.' if @show_progress && @log_posts != :all