new stuff to work with cvs provenance and branches

- put changesets in order, create new commitids for them by storing
a temporary commitid in CVSROOT/commitids-{tree}, running 'cvs
show' on it, reading the output hash, then overwrite the hash in
the commitids file with what we just got, then finally write out
all those new commitids to every revision

- add some new hacks for specific files in src/ that have revisions
with missing previous revisions, and some that are obviously part
of changesets with existing commitids but don't have them for some
reason

- sprinkle some BEGIN/COMMIT around heavy sqlite ops for speed, fix
index names so they're actually unique :/

- be more careful about string encoding in sqlite, get everything to
utf-8. data from rlog was coming in as binary, now iso-8859-1,
and needs to be properly converted to utf-8 before going into
sqlite, otherwise later queries trying to match on string data
(which are probably using utf-8 strings) will fail

- work better when re-running like not complaining about cvs-tmp
existing, since this is no longer intended to just run once.
store cheap cksum of file to detect changes to speed up re-scan.

README.md

···
 ###OpenBSD commitid generator
 
-A work in progress to assign `commitid` identifiers to all files in OpenBSD's
-CVS trees for commits before `commitid` functionality was enabled.
+A work in progress to assign CVS provenance-style `commitid` identifiers to
+all revisions of all files in OpenBSD's CVS trees.
 
 ####Usage
 
···
 
        `$ ruby openbsd-commitid.rb`
 
-**NOTE**: This script relies on recently added changes to OpenBSD's `rlog`
-and `cvs` tools:
+**NOTE**: This script relies on recently added changes to OpenBSD's `rlog` and
+`cvs` tools:
 
 - `cvs admin -C` to set a revision's `commitid`
-- `rlog -E` and `rlog -S` to control the revision separators in `rlog`
-  output, since the default line of dashes appears in old commit messages
-
-####Details
-
-This script does the following steps for each of the `src`, `ports`, `www`,
-and `xenocara` trees.
+- `rlog -E` and `rlog -S` to control the revision separators in `rlog` output,
+  since the default line of dashes appears in old commit messages
 
-1. Recurse a directory of RCS ,v files (`/var/cvs-commitid/`), create a
-"files" record for each one.
-
-2. Run `rlog` on each RCS file, parse the output and create a "revisions"
-record with each revision's author, date, version, commitid (if present), and
-log message.  Update the "files" record to note the first non-`dead` version
-in the file.
-
-3. Fetch all revisions not already matched to a changeset, ordered by author
-then date, and bundle them into changesets.  Create a new "changesets" record
-for each, then update each of those "revisions" records with the new changeset
-id.  By sorting all commits by author and date, it's possible to accurately
-find all files touched by an author in the same commit window.
-
-4. For each newly created "changesets" record, update them with a definitive
-timestamp, log message, author, and commitid (creating a new one if needed)
-based on all of the "revisions" with that changeset id.
-
-5. Do a `cvs checkout` from `/var/cvs-commitid/` to a temporary directory
-created in `/var/cvs-tmp/`, checking out revision 1.1 of each file.  For each
-file that has a first-non-`dead` version number that is not 1.1, do another
-`cvs checkout` of that version of the file so that every file in the tree is
-now present in the working checked-out tree.  This is required to operate on
-deleted files (moved to the Attic) and files where version 1.1 does not exist,
-such as those created on branches.
-
-6. For each "revisions" record of each "files" record that doesn't have a
-recorded `commitid` (meaning this script generated one while bundling it into a
-"changesets" record), run `cvs admin -C` to assign the `commitid` to that
-revision in the RCS `,v` file in `/var/cvs-commitid/`.
-
-7. `rm -rf` the temporary checked-out directory in `/var/cvs-tmp/` since the
-changes are now all present in the `/var/cvs-commitid/` tree.
+For details of how this script works, read `openbsd-commitid.rb`.

lib/db.rb

···
 class Db
   def initialize(dbf)
     @db = SQLite3::Database.new dbf
+    @db.results_as_hash = true
 
     @db.execute "CREATE TABLE IF NOT EXISTS changesets
       (id INTEGER PRIMARY KEY, date INTEGER, author TEXT, commitid TEXT,
-      log TEXT)"
-    @db.execute "CREATE UNIQUE INDEX IF NOT EXISTS u_commitid ON changesets
+      log TEXT, branch TEXT, csorder INTEGER)"
+    @db.execute "CREATE UNIQUE INDEX IF NOT EXISTS u_cs_commitid ON changesets
       (commitid)"
+    @db.execute "CREATE UNIQUE INDEX IF NOT EXISTS u_cs_csorder ON changesets
+      (csorder)"
+    @db.execute "CREATE INDEX IF NOT EXISTS cs_branch ON changesets (branch)"
 
     @db.execute "CREATE TABLE IF NOT EXISTS files
       (id INTEGER PRIMARY KEY, file TEXT, first_undead_version TEXT,
-      size INTEGER)"
-    @db.execute "CREATE UNIQUE INDEX IF NOT EXISTS u_file ON files
+      cksum TEXT)"
+    @db.execute "CREATE UNIQUE INDEX IF NOT EXISTS u_f_file ON files
       (file)"
 
     @db.execute "CREATE TABLE IF NOT EXISTS revisions
       (id INTEGER PRIMARY KEY, file_id INTEGER, changeset_id INTEGER,
       date INTEGER, version TEXT, author TEXT, commitid TEXT, log TEXT,
-      state TEXT)"
-    @db.execute "CREATE UNIQUE INDEX IF NOT EXISTS u_revision ON revisions
+      state TEXT, branch TEXT)"
+    @db.execute "CREATE UNIQUE INDEX IF NOT EXISTS u_r_revision ON revisions
       (file_id, version)"
-    @db.execute "CREATE INDEX IF NOT EXISTS empty_changesets ON revisions
+    @db.execute "CREATE INDEX IF NOT EXISTS r_empty_changesets ON revisions
       (changeset_id)"
-    @db.execute "CREATE INDEX IF NOT EXISTS cs_by_commitid ON revisions
+    @db.execute "CREATE INDEX IF NOT EXISTS r_cs_by_commitid ON revisions
       (commitid, changeset_id)"
-    @db.execute "CREATE INDEX IF NOT EXISTS all_revs_by_author ON revisions
+    @db.execute "CREATE INDEX IF NOT EXISTS r_all_revs_by_author ON revisions
       (author, date)"
-    @db.execute "CREATE INDEX IF NOT EXISTS all_revs_by_version_and_state ON
+    @db.execute "CREATE INDEX IF NOT EXISTS r_all_revs_by_version_and_state ON
       revisions (version, state)"
+    @db.execute "CREATE INDEX IF NOT EXISTS r_branch ON revisions (branch)"
 
-    @db.results_as_hash = true
+    @db.execute("CREATE TABLE IF NOT EXISTS vendor_branches
+      (id INTEGER PRIMARY KEY, revision_id INTEGER, branch TEXT)")
+    @db.execute("CREATE INDEX IF NOT EXISTS vb_revision ON vendor_branches
+      (revision_id)")
+    @db.execute("CREATE INDEX IF NOT EXISTS vb_branch_branch ON vendor_branches
+      (branch)")
   end
 
   def execute(*args)
···
     else
       @db.execute(*args)
     end
+  end
+
+  def last_insert_row_id
+    @db.last_insert_row_id
   end
 end

lib/outputter.rb

···
     printlog = Proc.new {
       fh.puts "Changes by:     #{last["author"]}@#{domain}   " <<
         Time.at(last["date"].to_i).strftime("%Y/%m/%d %H:%M:%S")
-      fh.puts "Commitid:       #{last["commitid"]}"
+      if last["commitid"].to_s != ""
+        fh.puts "Commitid:       #{last["commitid"]}"
+      end
+      if last["branch"].to_s != ""
+        fh.puts "Branch:         #{last["branch"]}"
+      end
       fh.puts ""
       fh.puts "Modified files:"
 
···
     }
 
     @scanner.db.execute("SELECT
-    changesets.date, changesets.author, changesets.commitid, changesets.log,
-    files.file
+    changesets.csorder, changesets.date, changesets.author,
+    changesets.commitid, changesets.log, files.file, revisions.branch
     FROM changesets
     LEFT OUTER JOIN revisions ON revisions.changeset_id = changesets.id
     LEFT OUTER JOIN files ON revisions.file_id = files.id
-    ORDER BY changesets.date, files.file") do |csfile|
-      if csfile["commitid"] == last["commitid"]
+    ORDER BY changesets.csorder, files.file") do |csfile|
+      if csfile["csorder"] == last["csorder"]
         files.push csfile["file"]
       else
         if files.any?

lib/rcsfile.rb

···
 #
 
 class RCSFile
-  attr_accessor :revisions, :first_undead_version
+  attr_accessor :file, :revisions, :symbols, :first_undead_version
 
   RCSEND = "==================OPENBSD_COMMITID_RCS_END=================="
   REVSEP = "------------------OPENBSD_COMMITID_REV_SEP------------------"
 
   def initialize(file)
+    @file = file
     @revisions = {}
+    @symbols = {}
 
     blocks = []
     IO.popen([ "rlog", "-E#{RCSEND}", "-S#{REVSEP}", file ]) do |rlog|
-      blocks = rlog.read.force_encoding("binary").
+      blocks = rlog.read.force_encoding("iso-8859-1").
         split(/^(#{REVSEP}|#{RCSEND})\n?$/).
         reject{|b| b == RCSEND || b == REVSEP }
     end
···
       raise "file #{file} didn't come out of rlog properly"
     end
 
-    blocks.shift
+    insymbols = false
+    blocks.shift.split("\n").each do |l|
+      if l.match(/^symbolic names:/)
+        insymbols = true
+      elsif insymbols && (m = l.match(/^\t(.+): ([\d\.]+)$/))
+        @symbols[m[1].encode("UTF-8")] = m[2].encode("UTF-8")
+      else
+        insymbols = false
+      end
+    end
+
     blocks.each do |block|
-      rev = RCSRevision.new(block)
+      rev = RCSRevision.new(self, block)
       if @revisions[rev.version]
         raise "duplicate revision #{rev.version} in #{file}"
       end
···
       # this has nothing to do with Gem, but it has a version comparator
       sort{|a,b| Gem::Version.new(a.version) <=> Gem::Version.new(b.version) }.
       select{|r| r.state != "dead" }.first.version
+  end
+
+  def to_s
+    "RCSFile: #{@file}"
   end
 end

lib/rcsrevision.rb

···
 require "date"
 
 class RCSRevision
-  attr_accessor :version, :date, :author, :state, :lines, :commitid, :log
+  attr_accessor :rcsfile, :version, :date, :author, :state, :lines, :commitid,
+    :log, :branch, :vendor_branches
+
+  def self.previous_of(ver)
+    nums = ver.split(".").map{|z| z.to_i }
+
+    if nums.last == 1
+      # 1.3.2.1 -> 1.3
+      2.times { nums.pop }
+    else
+      # 1.3.2.2 -> 1.3.2.1
+      nums[nums.count - 1] -= 1
+    end
+
+    outnum = nums.join(".")
+    if outnum == ""
+      return "0"
+    else
+      return outnum
+    end
+  end
+
+  # 1.1.0.2 -> 1.1.2.1
+  def self.first_branch_version_of(ver)
+    nums = ver.split(".").map{|z| z.to_i }
+
+    if nums[nums.length - 2] != 0
+      return ver
+    end
+
+    last = nums.pop
+    nums.pop
+    nums.push last
+    nums.push 1
+
+    return nums.join(".")
+  end
+
+  def self.is_vendor_branch?(ver)
+    !!ver.match(/^1\.1\.1\..*/)
+  end
+
+  def self.is_trunk?(ver)
+    ver.split(".").count == 2
+  end
 
   # str: "revision 1.7\ndate: 1996/12/14 12:17:33;  author: mickey;  state: Exp;  lines: +3 -3;\n-Wall'ing."
-  def initialize(str)
+  def initialize(rcsfile, str)
+    @rcsfile = rcsfile
     @version = nil
     @date = 0
     @author = nil
···
     @lines = nil
     @commitid = nil
     @log = nil
+    @branch = nil
+    @vendor_branches = []
 
     lines = str.gsub(/^\s*/, "").split("\n")
     # -> [
···
       lines.delete_at(2)
     end
 
-    @version = lines.first.scan(/^revision ([\d\.]+)($|\tlocked by)/).first.first
+    @version = lines.first.scan(/^revision ([\d\.]+)($|\tlocked by)/).first.
+      first.encode("UTF-8")
     # -> "1.7"
 
     # date/author/state/lines/commitid line
     lines[1].split(/;[ \t]*/).each do |piece|
       kv = piece.split(": ")
-      self.send(kv[0] + "=", kv[1])
+      self.send(kv[0] + "=", kv[1].encode("UTF-8"))
     end
     # -> @date = "1996/12/14 12:17:33", @author = "mickey", ...
 
···
     end
     # -> @date = 850565853
 
-    @log = lines[2, lines.count].join("\n")
+    @log = lines[2, lines.count].join("\n").encode("UTF-8",
+      :invalid => :replace, :undef => :replace, :replace => "?")
+
+    if @version.match(/^\d+\.\d+$/)
+      # no branch
+    elsif @version.match(/^1\.1\.1\./) ||
+    (@version == "1.1.2.1" && @branch == nil)
+      # vendor
+      @rcsfile.symbols.each do |k,v|
+        if v == "1.1.1"
+          @vendor_branches.push k
+        end
+      end
+    elsif m = @version.match(/^(\d+)\.(\d+)\.(\d+)\.\d+$/)
+      # 1.2.2.3 -> 1.2.0.2
+      sym = [ m[1], m[2], "0", m[3] ].join(".")
+      @rcsfile.symbols.each do |s,v|
+        if v == sym
+          if @branch
+            raise "version #{@version} matched two symbols (#{@branch}, #{s})"
+          end
+
+          @branch = s
+        end
+      end
+
+      if !@branch && @rcsfile.symbols.values.include?(@version)
+        # if there's an exact match, this was probably just an import done with
+        # a vendor branch id (import -b)
+      elsif !@branch
+        # branch was deleted, but we don't want this appearing on HEAD, so call
+        # it something
+        @branch = "_branchless_#{@version.gsub(".", "_")}"
+      end
+
+      if @branch && @rcsfile.symbols[@branch] &&
+      @rcsfile.symbols[@branch].match(/^1\.1\.0\.\d+$/)
+        # this is also a vendor branch
+        if !@vendor_branches.include?(@branch)
+          @vendor_branches.push @branch
+        end
+      end
+    else
+      raise "TODO: handle version #{@version}"
+    end
   end
 end

lib/scanner.rb

···
 #
 
 class Scanner
-  attr_accessor :outputter, :db
+  attr_accessor :outputter, :db, :commitid_hacks, :prev_revision_hacks
 
   # how long commits by the same author with the same commit message can be
   # from each other and still be grouped in the same changeset
···
     @db = Db.new dbf
     @root = (root + "/").gsub(/\/\//, "/")
     @outputter = Outputter.new(self)
+    @prev_revision_hacks = {}
+    @commitid_hacks = {}
   end
 
   def recursively_scan(dir = nil)
···
   end
 
   def scan(f)
-    stat = File.stat(f)
+    cksum = ""
+    IO.popen([ "cksum", "-q", f ]) do |c|
+      parts = c.read.force_encoding("iso-8859-1").split(" ")
+      if parts.length != 2
+        raise "invalid output from cksum: #{parts.inspect}"
+      end
+
+      cksum = parts[0].encode("utf-8")
+    end
+
     canfile = f[@root.length, f.length - @root.length].gsub(/(^|\/)Attic\//,
       "/").gsub(/^\/*/, "")
 
-    fid = @db.execute("SELECT id, first_undead_version, size FROM files " +
+    fid = @db.execute("SELECT id, first_undead_version, cksum FROM files " +
       "WHERE file = ?", [ canfile ]).first
-    if fid && fid["size"].to_i > 0 && fid["size"].to_i == stat.size
+    if fid && fid["cksum"].to_s == cksum
       return
     end
 
···
 
     rcs = RCSFile.new(f)
 
+    @db.execute("BEGIN")
+
     if fid
       if fid["first_undead_version"] != rcs.first_undead_version
         @db.execute("UPDATE files SET first_undead_version = ? WHERE id = ?",
···
     end
     raise if !fid
 
+    if @commitid_hacks && @commitid_hacks[canfile]
+      @commitid_hacks[canfile].each do |v,cid|
+        if rcs.revisions[v].commitid &&
+        rcs.revisions[v].commitid != cid
+          raise "hack for #{canfile}:#{v} commitid of #{cid.inspect} would " +
+            "overwrite #{rcs.revisions[v].commitid}"
+        end
+
+        puts " faking commitid for revision #{v} -> #{cid}"
+        rcs.revisions[v].commitid = cid
+      end
+    end
+
     rcs.revisions.each do |r,rev|
       rid = @db.execute("SELECT id, commitid FROM revisions WHERE " +
         "file_id = ? AND version = ?", [ fid["id"], r ]).first
···
             "AND version = ?", [ rev.commitid, fid["id"], rev.version ])
         end
       else
-        puts "  inserted #{r}, authored #{rev.date} by #{rev.author}" +
+        # files added on branches/imports have unhelpful commit messages with
+        # the helpful ones on the branch versions, so copy them over while
+        # we're here
+        if rev.log.to_s == "Initial revision"
+          if r == "1.1" && rcs.revisions["1.1.1.1"]
+            rev.log = rcs.revisions["1.1.1.1"].log
+            puts "  revision #{r} using log from 1.1.1.1"
+          else
+            puts "  revision #{r} keeping log #{rev.log.inspect}, no 1.1.1.1"
+          end
+        elsif m = rev.log.to_s.
+        match(/\Afile .+? was initially added on branch ([^\.]+)\.\z/)
+          brver = nil
+          if br = rcs.symbols[m[1]]
+            brver = RCSRevision.first_branch_version_of(br)
+            if !rcs.revisions[brver]
+              if rcs.revisions[brver + ".1"]
+                brver += ".1"
+              else
+                puts "  revision #{r} keeping log #{rev.log.inspect}, no #{brver}"
+                brver = nil
+              end
+            end
+          end
+
+          if brver
+            rev.log = rcs.revisions[brver].log
+            puts "  revision #{r} using log from #{brver}"
+
+            # but consider this trunk revision on the branch the file was added
+            # on, just so we keep it in the same changeset
+            rev.branch = rcs.revisions[brver].branch
+          else
+            puts "  revision #{r} keeping log #{rev.log.inspect}, no #{m[1]}"
+          end
+        end
+
+        puts "  inserted #{r}" +
+          (rev.branch ? " (branch #{rev.branch})" : "") +
+          ", authored #{rev.date} by #{rev.author}" +
           (rev.commitid ? ", commitid #{rev.commitid}" : "")
 
         @db.execute("INSERT INTO revisions (file_id, date, version, author, " +
-          "commitid, state, log) VALUES (?, ?, ?, ?, ?, ?, ?)",
+          "commitid, state, log, branch) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
           [ fid["id"], rev.date, rev.version, rev.author, rev.commitid,
-          rev.state, rev.log ])
+          rev.state, rev.log, rev.branch ])
+        rid = { "id" => @db.last_insert_row_id }
       end
-    end
 
-    @db.execute("UPDATE files SET size = ? WHERE id = ?",
-      [ stat.size, fid["id"] ])
-  end
+      vbs = @db.execute("SELECT branch FROM vendor_branches WHERE " +
+        "revision_id = ?", [ rid["id"] ]).map{|r| r["branch"] }.flatten
 
-  def stray_commitids_to_changesets
-    stray_commitids = @db.execute("SELECT DISTINCT author, commitid FROM " +
-      "revisions WHERE commitid IS NOT NULL AND changeset_id IS NULL")
-    stray_commitids.each do |row|
-      csid = @db.execute("SELECT id FROM changesets WHERE commitid = ?",
-        [ row["commitid"] ]).first
-      if !csid
-        @db.execute("INSERT INTO changesets (author, commitid) VALUES (?, ?)",
-          [ row["author"], row["commitid"] ])
-        csid = @db.execute("SELECT id FROM changesets WHERE commitid = ?",
-          [ row["commitid"] ]).first
+      rev.vendor_branches.each do |vb|
+        if !vbs.include?(vb)
+          puts "   inserting vendor branch #{vb}"
+          @db.execute("INSERT INTO vendor_branches (revision_id, branch) " +
+            "VALUES (?, ?)", [ rid["id"], vb ])
+        end
       end
-      raise if !csid
+
+      vbs.each do |vb|
+        if !rev.vendor_branches.include?(vb)
+          @db.execute("DELETE FROM vendor_branches WHERE revision_id = ? " +
+            "AND branch = ?", [ rid["id"], vb ])
+        end
+      end
+    end
 
-      puts "commitid #{row["commitid"]} -> changeset #{csid["id"]}"
+    @db.execute("UPDATE files SET cksum = ? WHERE id = ?",
+      [ cksum, fid["id"] ])
 
-      @db.execute("UPDATE revisions SET changeset_id = ? WHERE commitid = ?",
-        [ csid["id"], row["commitid"] ])
-    end
+    @db.execute("COMMIT")
   end
 
   def group_into_changesets
+    puts "grouping into changesets"
+
     new_sets = []
     last_row = {}
     cur_set = []
 
-    # TODO: don't conditionalize with null changeset_ids, to allow this to run
-    # incrementally and match new commits to old changesets
+    @db.execute("BEGIN")
+
+    # commits by the same author with the same log message within a small
+    # timeframe are grouped together
     @db.execute("SELECT * FROM revisions WHERE changeset_id IS NULL ORDER " +
-    "BY author ASC, date ASC") do |row|
-      # commits by the same author with the same log message (unless they're
-      # initial imports - 1.1.1.1) within a small timeframe are grouped
-      # together
-      if last_row.any? && row["author"] == last_row["author"] &&
-      (row["log"] == last_row["log"] || row["log"] == "Initial revision" ||
-      last_row["log"] == "Initial revision") &&
+    "BY author ASC, branch ASC, commitid ASC, date ASC") do |row|
+      if last_row.any? &&
+      row["author"] == last_row["author"] &&
+      row["branch"] == last_row["branch"] &&
+      row["log"] == last_row["log"] &&
+      row["commitid"] == last_row["commitid"] &&
       row["date"].to_i - last_row["date"].to_i <= MAX_GROUP_WINDOW
         cur_set.push row["id"].to_i
       elsif !last_row.any?
···
     end
 
     new_sets.each do |s|
-      puts "new set with revision ids #{s.inspect}"
+      puts " new set with revision ids #{s.inspect}"
       @db.execute("INSERT INTO changesets (id) VALUES (NULL)")
       id = @db.execute("SELECT last_insert_rowid() AS id").first["id"]
       raise if !id
···
     if @db.execute("SELECT * FROM revisions WHERE changeset_id IS NULL").any?
       raise "still have revisions with empty changesets"
     end
+
+    @db.execute("COMMIT")
+  end
+
+  def stray_commitids_to_changesets
+    @db.execute("BEGIN")
+
+    puts "finding stray commitids"
+
+    stray_commitids = @db.execute("SELECT DISTINCT author, commitid FROM " +
+      "revisions WHERE commitid IS NOT NULL AND changeset_id IS NULL")
+    stray_commitids.each do |row|
+      csid = @db.execute("SELECT id FROM changesets WHERE commitid = ?",
+        [ row["commitid"] ]).first
+      if !csid
+        @db.execute("INSERT INTO changesets (author, commitid) VALUES (?, ?)",
+          [ row["author"], row["commitid"] ])
+        csid = @db.execute("SELECT id FROM changesets WHERE commitid = ?",
+          [ row["commitid"] ]).first
+      end
+      raise if !csid
+
+      puts " commitid #{row["commitid"]} -> changeset #{csid["id"]}"
+
+      @db.execute("UPDATE revisions SET changeset_id = ? WHERE commitid = ?",
+        [ csid["id"], row["commitid"] ])
+    end
+
+    @db.execute("COMMIT")
   end
 
   def fill_in_changeset_data
+    puts "assigning dates to changesets"
+
+    @db.execute("BEGIN")
+
     cses = {}
     @db.execute("SELECT id, commitid FROM changesets WHERE date IS NULL") do |c|
       cses[c["id"]] = c["commitid"]
     end
 
+    # create canonical dates for each changeset, so we can pull them back out
+    # in order
     cses.each do |csid,comid|
       date = nil
       commitid = comid
       log = nil
       author = nil
+      branch = nil
 
       @db.execute("SELECT * FROM revisions WHERE changeset_id = ? ORDER BY " +
       "date ASC", [ csid ]) do |rev|
···
           date = rev["date"]
         end
 
-        if rev["log"] != "Initial revision"
+        if log && rev["log"] != log
+          raise "logs different between revs of #{csid}"
+        else
           log = rev["log"]
         end
 
···
         else
           author = rev["author"]
         end
-      end
 
-      if commitid.to_s == ""
-        commitid = ""
-        while commitid.length < 16
-          c = rand(75) + 48
-          if ((c >= 48 && c <= 57) || (c >= 65 && c <= 90) ||
-          (c >= 97 && c <= 122))
-            commitid << c.chr
-          end
+        if branch && rev["branch"] != branch
+          raise "branches different between revs of #{csid}"
+        else
+          branch = rev["branch"]
         end
       end
 
···
         raise "no date for changeset #{csid}"
       end
 
-      puts "changeset #{csid} -> commitid #{commitid}"
+      @db.execute("UPDATE changesets SET date = ?, log = ?, author = ?, " +
+        "branch = ? WHERE id = ?", [ date, log, author, branch, csid ])
+    end
+
+    @db.execute("COMMIT")
 
-      @db.execute("UPDATE changesets SET date = ?, commitid = ?, log = ?, " +
-        "author = ? WHERE id = ?", [ date, commitid, log, author, csid ])
+    puts "assigning changeset order"
+
+    cses = []
+    @db.execute("SELECT id FROM changesets WHERE csorder IS NULL ORDER BY " +
+    "date, author") do |c|
+      cses.push c["id"]
     end
-  end
 
-  def repo_surgery(tmp_dir, cvs_root, tree)
-    puts "checking out #{tree} from #{cvs_root} to #{tmp_dir}"
+    highestcs = @db.execute("SELECT MAX(csorder) AS lastcs FROM changesets " +
+      "WHERE csorder IS NOT NULL").first["lastcs"].to_i
 
-    Dir.chdir(tmp_dir)
+    @db.execute("BEGIN")
+    cses.each do |cs|
+      highestcs += 1
+      @db.execute("UPDATE changesets SET csorder = ?, commitid = NULL WHERE " +
+        "id = ?", [ highestcs, cs ])
+    end
+    @db.execute("COMMIT")
+  end
 
+  def stage_tmp_cvs(tmp_dir, cvs_root, tree)
     # for a deleted file to be operated by with cvs admin, it must be
     # present in the CVS/Entries files, so check out all files at rev 1.1 so we
     # know they will not be deleted.  otherwise cvs admin will fail silently
-    system("cvs", "-Q", "-d", cvs_root, "co", "-r1.1", tree) ||
-      raise("cvs checkout returned non-zero")
+    if File.exists?("#{tmp_dir}/#{tree}/CVS/Entries")
+      puts "updating #{tmp_dir}#{tree} from #{cvs_root}"
+      Dir.chdir("#{tmp_dir}/#{tree}")
+      system("cvs", "-Q", "-d", cvs_root, "update", "-PAd", "-r1.1") ||
+        raise("cvs update returned non-zero")
+    else
+      puts "checking out #{cvs_root}#{tree} to #{tmp_dir}"
+      Dir.chdir(tmp_dir)
+      system("cvs", "-Q", "-d", cvs_root, "co", "-r1.1", tree) ||
+        raise("cvs checkout returned non-zero")
+    end
+
+    Dir.chdir(tmp_dir)
 
     # but if any files were added on a branch or somehow have a weird history,
     # their 1.1 revision will be dead so check out any non-dead revision of
···
     @db.execute("SELECT
     file, first_undead_version
     FROM files
-    WHERE first_undead_version NOT LIKE '1.1'") do |rev|
+    WHERE first_undead_version NOT LIKE '1.1' AND
+    id IN (SELECT file_id FROM revisions WHERE commitid IS NULL)") do |rev|
       dead11s[rev["file"]] = rev["first_undead_version"]
     end
 
···
         "#{tree}/#{confile}") ||
         raise("cvs co -r#{rev} #{confile} failed")
     end
+
+    Dir.chdir("#{tmp_dir}/#{tree}")
+  end
+
+  def recalculate_commitids(tmp_dir, cvs_root, tree, genesis)
     Dir.chdir(tmp_dir + "/#{tree}")
 
-    csid = nil
-    @db.execute("SELECT
-    files.file, changesets.commitid, changesets.author, changesets.date,
-    revisions.version
+    puts "recalculating new commitids from genesis #{genesis}"
+
+    gfn = "#{cvs_root}/CVSROOT/commitid_genesis"
+    if File.exists?(gfn) && File.read(gfn).strip != genesis
+      raise "genesis in #{gfn} is not #{genesis.inspect}"
+    else
+      File.write("#{cvs_root}/CVSROOT/commitid_genesis", genesis + "\n")
+    end
+
+    changesets = []
+    @db.execute("SELECT id, csorder, commitid FROM changesets
+    ORDER BY csorder ASC") do |cs|
+      changesets.push cs
+    end
+
+    puts " writing commitids-#{tree} (#{changesets.length} " +
+      "changeset#{changesets.length == 1 ? "" : "s"})"
+
+    commitids = File.open("#{cvs_root}/CVSROOT/commitids-#{tree}", "w+")
+
+    # every changeset needs to know the revisions of its files from the
+    # previous change, taking into account branches.  we can easily calculate
+    # this, but we should make sure that calculated revision actually exists
+    files = {}
+    @db.execute("SELECT id, file FROM files") do |row|
+      files[row["id"]] = row["file"]
+    end
+    files.each do |id,file|
+      vers = []
+
+      @db.execute("SELECT version FROM revisions WHERE file_id = ?",
+      [ id ]) do |rev|
+        vers.push rev["version"]
+      end
+
+      vers.each do |rev|
+        if prev_revision_hacks[file] && (hpre = prev_revision_hacks[file][rev])
+          puts " faking previous revision of #{file} #{rev} -> #{hpre}"
+          pre = hpre
+        else
+          pre = RCSRevision.previous_of(rev)
+        end
+
+        if pre != "0" && !vers.include?(pre)
+          raise "#{file}: revision #{rev} previous #{pre} not found"
+        end
+      end
+    end
+    files = {}
+
+    # for each changeset with no commitid, store it in the commitids-* file
+    # with a temporary commitid of just its changeset number, do a 'cvs show'
+    # on it to calculate the actual commitid, then overwrite that hash in the
+    # commitids file, and store our new one
+    changesets.each do |cs|
+      cline = []
+      commitid = ""
+      if cs["commitid"].to_s != ""
+        commitid = cs["commitid"]
+      else
+        commitid = sprintf("01-%064d-%07d", cs["csorder"], cs["csorder"])
+      end
+
+      # order by length(revisions.version) to put 1.1 first, then 1.1.1.1, to
+      # match 'cvs import'
+      @db.execute("SELECT
+      files.file, revisions.version, revisions.branch
+      FROM revisions
+      LEFT OUTER JOIN files ON files.id = revisions.file_id
+      WHERE revisions.changeset_id = ?
+      ORDER BY files.file ASC, LENGTH(revisions.version) ASC,
+      revisions.version ASC", [ cs["id"] ]) do |rev|
+        if cline.length == 0
+          cline.push commitid
+        end
+
+        cline.push [ RCSRevision.previous_of(rev["version"]), rev["version"],
+          rev["branch"].to_s, rev["file"].gsub(/,v$/, "") ].join(":")
+      end
+
+      pos = commitids.pos
+      commitids.puts cline.join("\t")
+
+      if cs["commitid"].to_s == ""
+        commitids.fsync
+
+        newcsum = `cvs show #{commitid} | tail -n +2 | cksum -a sha512/256`.strip
+        if $?.exitstatus != 0
+          raise "failed running cvs show #{commitid}"
+        end
+
+        # null
+        if newcsum == "c672b8d1ef56ed28ab87c3622c5114069bdd3ad7b8f9737498d0c01ecef0967a"
+          raise "failed getting new commitid from #{commitid}"
+        end
+
+        newid = sprintf("01-%64s-%07d", newcsum, cs["csorder"])
+
+        @db.execute("UPDATE changesets SET commitid = ? WHERE id = ?",
+          [ newid, cs["id"] ])
+
+        puts " changeset #{cs["csorder"]} -> #{newid}"
+
+        # go back, rewrite just our commitid, then get ready for the next line
+        commitids.seek(pos)
+        commitids.write(newid)
+        commitids.seek(0, IO::SEEK_END)
+        commitids.fsync
+      else
+        puts " changeset #{cs["csorder"]} == #{cs["commitid"]}"
+      end
+    end
+
+    commitids.close
+  end
+
+  def repo_surgery(tmp_dir, cvs_root, tree)
+    puts "updating commitids in rcs files at #{cvs_root} via #{tmp_dir}"
+
+    Dir.chdir("#{tmp_dir}/#{tree}")
+
+    # for each revision we have in the db (picked up from a scan) that has a
+    # different commitid from what we assigned to its changeset, update the
+    # commitid in the rcs file in the repo, and then our revisions records
+    @db.execute("
+    SELECT
+    files.file, changesets.commitid, revisions.version, revisions.id AS revid,
+    revisions.commitid AS revcommitid
     FROM revisions
-    LEFT OUTER JOIN files ON files.id = file_id
+    LEFT OUTER JOIN files ON files.id = revisions.file_id
     LEFT OUTER JOIN changesets ON revisions.changeset_id = changesets.id
-    WHERE revisions.commitid IS NULL
+    WHERE changesets.commitid != IFNULL(revisions.commitid, '')
     ORDER BY changesets.date ASC, files.file ASC") do |rev|
-      if csid == nil || rev["commitid"] != csid
-        puts " commit #{rev["commitid"]} at #{Time.at(rev["date"])} by " +
-          rev["author"]
-        csid = rev["commitid"]
-      end
-
-      puts "  #{rev["file"]} #{rev["version"]}"
+      puts [ "", rev["file"], rev["version"], rev["revcommitid"], "->",
+        rev["commitid"] ].join(" ")
 
       output = nil
       IO.popen(ca = [ "cvs", "admin", "-C",
···
       end
     end
 
-    puts "cleaning up #{tmp_dir}/#{tree}"
-
-    system("rm", "-rf", tmp_dir + "/#{tree}") ||
-      raise("rm of #{tmp_dir}/#{tree} failed")
+    # re-read commitids and update file checksums since we probably just
+    # changed many of them, which will then update commitids in revisions table
+    sc.recursively_scan
   end
 end

openbsd-commitid.rb

···
 # THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 #
 
-DIR = File.dirname(__FILE__) + "/lib/"
+PWD = File.dirname(__FILE__)
 
-require DIR + "db"
-require DIR + "scanner"
-require DIR + "rcsfile"
-require DIR + "rcsrevision"
-require DIR + "outputter"
+require PWD + "/lib/db"
+require PWD + "/lib/scanner"
+require PWD + "/lib/rcsfile"
+require PWD + "/lib/rcsrevision"
+require PWD + "/lib/outputter"
 
 CVSROOT = "/var/cvs-commitid/"
 CVSTMP = "/var/cvs-tmp/"
 CVSTREES = [ "src", "ports", "www", "xenocara" ]
 
+GENESIS = "01-f96d46480b33dcec5924884fef54166e169fc08d19f1d1812f5cd2d1f704219a-0000000"
+
 CVSTREES.each do |tree|
-  if Dir.exists?("#{CVSTMP}/#{tree}/CVS")
-    raise "clean out #{CVSTMP} first"
+  if !Dir.exists?("#{CVSROOT}/#{tree}")
+    next
   end
-end
 
-PWD = Dir.pwd
+  sc = Scanner.new(PWD + "/db/openbsd-#{tree}.db", "#{CVSROOT}/#{tree}/")
 
-CVSTREES.each do |tree|
-  sc = Scanner.new(PWD + "/db/openbsd-#{tree}.db", "#{CVSROOT}/#{tree}/")
+  if tree == "src"
+    # these revisions didn't get proper commitids with the others in the
+    # changeset, so fudge them
+    sc.commitid_hacks = {
+      "sys/dev/pv/xenvar.h,v" => {
+        "1.1" => "Ij2SOB19ATTH0yEx",
+        "1.2" => "pq3FAYuwXteAsF4d",
+        "1.3" => "C8vFI0RNH9XPJUKs",
+      },
+      "usr.bin/mg/theo.c,v" => {
+        "1.144" => "gSveQVkxMLs6vRqK",
+        "1.145" => "GbEBL4CfPvDkB8hj",
+        "1.146" => "8rkHsVfUx5xgPXRB",
+      },
+    }
+
+    # some rcs files have manually edited history that we need to work around
+    sc.prev_revision_hacks = {
+      # initial history gone?
+      "sbin/isakmpd/pkcs.c,v" => { "1.4" => "0" },
+      # 1.6 gone
+      "sys/arch/sun3/sun3/machdep.c,v" => { "1.7" => "1.5" },
+    }
+  end
+
+  # walk the directory of RCS files, create a "files" record for each one,
+  # then run `rlog` on it and create a "revisions" record for each
   sc.recursively_scan
+
+  # group revisions into changesets by date/author/message, or for newer
+  # commits, their stored commitid
   sc.group_into_changesets
+
+  # make sure every revision is accounted for
   sc.stray_commitids_to_changesets
-  sc.fill_in_changeset_data
 
-  sc.repo_surgery(CVSTMP, CVSROOT, tree)
+  # assign a canonical date/message/order to each changeset
+  sc.fill_in_changeset_data
 
-  sc.outputter.changelog("cvs.openbsd.org",
-    f = File.open("out/Changelog-#{tree}", "w+"))
-  f.close
+  # check out the cvs tree in CVSTMP/tree and place each dead-1.1 file at its
+  # initial non-dead revision found during `rlog`
+  sc.stage_tmp_cvs(CVSTMP, CVSROOT, tree)
 
-  sc.outputter.history(f = File.open("out/history-#{tree}", "w+"))
-  f.close
+  # calculate a hash for each commit by running 'cvs show' on it, and store it
+  # in the commitids-{tree} file
+  sc.recalculate_commitids(CVSTMP, CVSROOT, tree, GENESIS)
 
-  sc.outputter.dup_script(f = File.open("out/add_commitids_to_#{tree}.sh",
-    "w+"), tree)
-  f.close
+  # and finally, update every revision of every file and write its calculated
+  # commitid, possibly replacing the random one already there
+  sc.repo_surgery(CVSTMP, CVSROOT, tree)
 end