2009-06-18 Eric Seidel <eric@webkit.org>
authoreric@webkit.org <eric@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 23 Jun 2009 08:40:22 +0000 (08:40 +0000)
committereric@webkit.org <eric@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 23 Jun 2009 08:40:22 +0000 (08:40 +0000)
        Reviewed by Dave Levin.

        WebKit needs a script to interact with bugzilla and automate
        parts of the patch posting and commit processes.
        https://bugs.webkit.org/show_bug.cgi?id=26283

        This is really a first-draft tool.
        It's to the point where it's useful to more people than just me now though.
        Git support works.  SVN support is written, but mostly untested.

        This tool requires BeautifulSoup and mechanize python modules to run:
        sudo easy_install BeautifulSoup
        sudo easy_install mechanize

        More important than the tool itself are the Bugzilla, Git and SVN class abstractions
        which I hope will allow easy writing of future tools.

        The tool currently implements 10 commands, described below.

        Helpers for scripting dealing with the commit queue:
        bugs-to-commit                 Bugs in the commit queue
        patches-to-commit              Patches attached to bugs in the commit queue

        Dealing with bugzilla:
        reviewed-patches BUGID         r+'d patches on a bug
        apply-patches BUGID            Applies all patches on a bug to the local working directory without committing.
        land-and-update BUGID          Lands the current working directory diff and updates the bug.
        land-patches [options] BUGID   Lands all patches on a bug optionally testing them first
        obsolete-attachments BUGID     Marks all attachments on a bug as obsolete.
        commit-message                 Prints a commit message suitable for the uncommitted changes.

        These effectively replace git-send-bugzilla:
        post-diff BUGID                Attaches the current working directory diff to a bug as a patch file.
        post-commits BUGID COMMITISH   Attaches a range of local commits to a bug as patch files.

        post-diff works for SVN and Git, post-commits only works for SCMs with local-commit support (like Git)

        land-* commands in a Git environment only work with simple patches due to svn-apply bugs:
        https://bugs.webkit.org/show_bug.cgi?id=26299
        https://bugs.webkit.org/show_bug.cgi?id=26300

        This script follows python style (similar to how for Obj-C we follow AppKit style)
        http://www.python.org/doc/essays/styleguide.html
        The Python community has a strong style culture and the WebKit style guide is silent re: Python.

        I've filed a bug to update the WebKit style guide to mention python:
        https://bugs.webkit.org/show_bug.cgi?id=26524

        * Scripts/bugzilla-tool: Added.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@44979 268f45cc-cd09-0410-ab3c-d52691b4dbfc

WebKitTools/ChangeLog
WebKitTools/Scripts/bugzilla-tool [new file with mode: 0755]
WebKitTools/Scripts/modules/__init__.py [new file with mode: 0644]
WebKitTools/Scripts/modules/bugzilla.py [new file with mode: 0644]
WebKitTools/Scripts/modules/scm.py [new file with mode: 0644]

index 67c74f5..a9ac25c 100644 (file)
@@ -1,3 +1,55 @@
+2009-06-18  Eric Seidel  <eric@webkit.org>
+
+        Reviewed by Dave Levin.
+
+        WebKit needs a script to interact with bugzilla and automate
+        parts of the patch posting and commit processes.
+        https://bugs.webkit.org/show_bug.cgi?id=26283
+
+        This is really a first-draft tool.
+        It's to the point where it's useful to more people than just me now though.
+        Git support works.  SVN support is written, but mostly untested.
+
+        This tool requires BeautifulSoup and mechanize python modules to run:
+        sudo easy_install BeautifulSoup
+        sudo easy_install mechanize
+
+        More important than the tool itself are the Bugzilla, Git and SVN class abstractions
+        which I hope will allow easy writing of future tools.
+
+        The tool currently implements 10 commands, described below.
+
+        Helpers for scripting dealing with the commit queue:
+        bugs-to-commit                 Bugs in the commit queue
+        patches-to-commit              Patches attached to bugs in the commit queue
+
+        Dealing with bugzilla:
+        reviewed-patches BUGID         r+'d patches on a bug
+        apply-patches BUGID            Applies all patches on a bug to the local working directory without committing.
+        land-and-update BUGID          Lands the current working directory diff and updates the bug.
+        land-patches [options] BUGID   Lands all patches on a bug optionally testing them first
+        obsolete-attachments BUGID     Marks all attachments on a bug as obsolete.
+        commit-message                 Prints a commit message suitable for the uncommitted changes.
+
+        These effectively replace git-send-bugzilla:
+        post-diff BUGID                Attaches the current working directory diff to a bug as a patch file.
+        post-commits BUGID COMMITISH   Attaches a range of local commits to a bug as patch files.
+
+        post-diff works for SVN and Git, post-commits only works for SCMs with local-commit support (like Git)
+
+        land-* commands in a Git environment only work with simple patches due to svn-apply bugs:
+        https://bugs.webkit.org/show_bug.cgi?id=26299
+        https://bugs.webkit.org/show_bug.cgi?id=26300
+
+        This script follows python style (similar to how for Obj-C we follow AppKit style)
+        http://www.python.org/doc/essays/styleguide.html
+        The Python community has a strong style culture and the WebKit style guide is silent re: Python.
+
+        I've filed a bug to update the WebKit style guide to mention python:
+        https://bugs.webkit.org/show_bug.cgi?id=26524
+
+        * Scripts/bugzilla-tool: Added.
+
 2009-06-22  Steve Falkenburg  <sfalken@apple.com>
 
         Remove errant line of code mistakenly checked in.
diff --git a/WebKitTools/Scripts/bugzilla-tool b/WebKitTools/Scripts/bugzilla-tool
new file mode 100755 (executable)
index 0000000..77b3f5f
--- /dev/null
@@ -0,0 +1,468 @@
+#!/usr/bin/python
+# Copyright (c) 2009, Google Inc. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+# 
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# A tool for automating dealing with bugzilla, posting patches, committing patches, etc.
+
+import os
+import re
+import subprocess
+import sys
+
+from optparse import OptionParser, IndentedHelpFormatter, SUPPRESS_USAGE, make_option
+
+# Import WebKit-specific modules.
+from modules.bugzilla import Bugzilla
+from modules.scm import detect_scm_system, ScriptError
+
+def log(string):
+    print >> sys.stderr, string
+
+def error(string):
+    log(string)
+    exit(1)
+
+# These could be put in some sort of changelogs.py.
+def latest_changelog_entry(changelog_path):
+    # e.g. 2009-06-03  Eric Seidel  <eric@webkit.org>
+    changelog_date_line_regexp = re.compile('^(\d{4}-\d{2}-\d{2})' # Consume the date.
+                                  + '\s+(.+)\s+' # Consume the name.
+                                  + '<([^<>]+)>$') # And finally the email address.
+    
+    entry_lines = []
+    changelog = open(changelog_path)
+    try:
+        log("Parsing ChangeLog: " + changelog_path)
+        # The first line should be a date line.
+        first_line = changelog.readline()
+        if not changelog_date_line_regexp.match(first_line):
+            return None
+        entry_lines.append(first_line)
+        
+        for line in changelog:
+            # If we've hit the next entry, return.
+            if changelog_date_line_regexp.match(line):
+                return ''.join(entry_lines)
+            entry_lines.append(line)
+    finally:
+        changelog.close()
+    # We never found a date line!
+    return None
+
+def modified_changelogs(scm):
+    changelog_paths = []
+    paths = scm.changed_files()
+    for path in paths:
+        if os.path.basename(path) == "ChangeLog":
+            changelog_paths.append(path)
+    return changelog_paths
+
+def commit_message_for_this_commit(scm):
+    changelog_paths = modified_changelogs(scm)
+    if not len(changelog_paths):
+        error("Found no modified ChangeLogs, can't create a commit message.")
+
+    changelog_messages = []
+    for path in changelog_paths:
+        changelog_entry = latest_changelog_entry(path)
+        if not changelog_entry:
+            error("Failed to parse ChangeLog: " + os.path.abspath(path))
+        changelog_messages.append(changelog_entry)
+    
+    # FIXME: We should sort and label the ChangeLog messages like commit-log-editor does.
+    return ''.join(changelog_messages)
+
+
+class Command:
+    def __init__(self, help_text, argument_names="", options=[]):
+        self.help_text = help_text
+        self.argument_names = argument_names
+        self.options = options
+        self.option_parser = OptionParser(usage=SUPPRESS_USAGE, add_help_option=False, option_list=self.options)
+    
+    def name_with_arguments(self, command_name):
+        usage_string = command_name
+        if len(self.options) > 0:
+            usage_string += " [options]"
+        if self.argument_names:
+            usage_string += " " + self.argument_names
+        return usage_string
+
+    def parse_args(self, args):
+        return self.option_parser.parse_args(args)
+
+    def execute(self, options, args, tool):
+        raise NotImplementedError, "subclasses must implement"
+
+
+class BugsInCommitQueue(Command):
+    def __init__(self):
+        Command.__init__(self, 'Bugs in the commit queue')
+
+    def execute(self, options, args, tool):
+        bug_ids = tool.bugs.fetch_bug_ids_from_commit_queue()
+        for bug_id in bug_ids:
+            print "%s" % tool.bugs.bug_url_for_bug_id(bug_id)
+
+
+class PatchesInCommitQueue(Command):
+    def __init__(self):
+        Command.__init__(self, 'Patches attached to bugs in the commit queue')
+
+    def execute(self, options, args, tool):
+        patches = tool.bugs.fetch_patches_from_commit_queue()
+        log("Patches in commit queue:")
+        for patch in patches:
+            print "%s" % patch['url']
+
+
+class ReviewedPatchesOnBug(Command):
+    def __init__(self):
+        Command.__init__(self, 'r+\'d patches on a bug', 'BUGID')
+
+    def execute(self, options, args, tool):
+        bug_id = args[0]
+        patches_to_land = tool.bugs.fetch_reviewed_patches_from_bug(bug_id)
+        for patch in patches_to_land:
+            print "%s" % patch['url']
+
+
+class ApplyPatchesFromBug(Command):
+    def __init__(self):
+        options = [
+            make_option("--no-update", action="store_false", dest="update", default=True, help="Don't update the working directory before applying patches"),
+            make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)"),
+            make_option("--no-clean", action="store_false", dest="clean", default=True, help="Don't check if the working directory is clean before applying patches"),
+        ]
+        Command.__init__(self, 'Applies all patches on a bug to the local working directory without committing.', 'BUGID', options=options)
+
+    def execute(self, options, args, tool):
+        bug_id = args[0]
+        patches = tool.bugs.fetch_reviewed_patches_from_bug(bug_id)
+        os.chdir(tool.scm().checkout_root)
+        if options.clean:
+            tool.scm().ensure_clean_working_directory(options.force_clean)
+        if options.update:
+            tool.scm().update_webkit()
+        
+        for patch in patches:
+            # FIXME: Should have an option to local-commit each patch after application.
+            tool.scm().apply_patch(patch)
+
+
+def bug_comment_from_commit_text(commit_text):
+    comment_lines = []
+    commit_lines = commit_text.splitlines()
+    for line in commit_lines:
+        comment_lines.append(line)
+        match = re.match("^Committed r(\d+)$", line)
+        if match:
+            revision = match.group(1)
+            comment_lines.append("http://trac.webkit.org/changeset/" + revision)
+            break
+    return "\n".join(comment_lines)
+
+
+class LandAndUpdateBug(Command):
+    def __init__(self):
+        Command.__init__(self, 'Lands the current working directory diff and updates the bug.', 'BUGID')
+
+    def execute(self, options, args, tool):
+        bug_id = args[0]
+        os.chdir(tool.scm().checkout_root)
+        commit_message = commit_message_for_this_commit(tool.scm())
+        commit_log = tool.scm().commit_with_message(commit_message)
+        comment_text = bug_comment_from_commit_text(commit_log)
+        tool.bugs.close_bug_as_fixed(bug_id, comment_text)
+
+
+class LandPatchesFromBug(Command):
+    def __init__(self):
+        options = [
+            make_option("--no-update", action="store_false", dest="update", default=True, help="Don't update the working directory before applying patches"),
+            make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)"),
+            make_option("--no-clean", action="store_false", dest="clean", default=True, help="Don't check if the working directory is clean before applying patches"),
+            make_option("--no-build", action="store_false", dest="build", default=True, help="Commit without building first, implies --no-test."),
+            make_option("--no-test", action="store_false", dest="test", default=True, help="Commit without runnning run-webkit-tests."),
+        ]
+        Command.__init__(self, 'Lands all patches on a bug optionally testing them first', 'BUGID', options=options)
+
+    @staticmethod
+    def run_and_throw_if_fail(script_name):
+        build_webkit_process = subprocess.Popen(script_name, shell=True)
+        return_code = build_webkit_process.wait()
+        if return_code:
+            raise ScriptError(script_name + " failed with code " + return_code)
+
+    def build_webkit(self):
+        self.run_and_throw_if_fail("build-webkit")
+
+    def run_webkit_tests(self):
+        self.run_and_throw_if_fail("run-webkit-tests")
+
+    def execute(self, options, args, tool):
+        bug_id = args[0]
+
+        try:
+            patches = tool.bugs.fetch_reviewed_patches_from_bug(bug_id)
+            commit_text = ""
+
+            os.chdir(tool.scm().checkout_root)
+            tool.scm().ensure_no_local_commits(options.force_clean)
+            if options.clean:
+                tool.scm().ensure_clean_working_directory(options.force_clean)
+            if options.update:
+                tool.scm().update_webkit()
+            
+            for patch in patches:
+                tool.scm().apply_patch(patch)
+                if options.build:
+                    self.build_webkit()
+                    if options.test:
+                        self.run_webkit_tests()
+                commit_message = commit_message_for_this_commit(tool.scm())
+                commit_log = tool.scm().commit_with_message(commit_message)
+                comment_text = bug_comment_from_commit_text(commit_log)
+                # If we're commiting more than one patch, update the bug as we go.
+                if len(patches) > 1:
+                    tool.bugs.obsolete_attachment(patch['id'], comment_text)
+
+            if len(patches) > 1:
+                commit_text = "All reviewed patches landed, closing."
+
+            tool.bugs.close_bug_as_fixed(bug_id, commit_text)
+        except ScriptError, error:
+            log(error)
+            # We could add a comment to the bug about the failure.
+
+
+class CommitMessageForCurrentDiff(Command):
+    def __init__(self):
+        Command.__init__(self, 'Prints a commit message suitable for the uncommitted changes.')
+
+    def execute(self, options, args, tool):
+        os.chdir(tool.scm().checkout_root)
+        print "%s" % commit_message_for_this_commit(tool.scm())
+
+
+class ObsoleteAttachmentsOnBug(Command):
+    def __init__(self):
+        Command.__init__(self, 'Marks all attachments on a bug as obsolete.', 'BUGID')
+
+    def execute(self, options, args, tool):
+        bug_id = args[0]
+        attachments = tool.bugs.fetch_attachments_from_bug(bug_id)
+        for attachment in attachments:
+            if not attachment['obsolete']:
+                tool.bugs.obsolete_attachment(attachment['id'])
+
+
+class PostDiffAsPatchToBug(Command):
+    def __init__(self):
+        options = [
+            make_option("--no-review", action="store_false", dest="review", default=True, help="Do not mark the patch for review."),
+            make_option("-m", "--description", action="store", type="string", dest="description", help="Description string for the attachment (default: 'patch')"),
+        ]
+        Command.__init__(self, 'Attaches the current working directory diff to a bug as a patch file.', 'BUGID', options=options)
+
+    def execute(self, options, args, tool):
+        bug_id = args[0]
+        
+        diff_process = subprocess.Popen(tool.scm().create_patch_command(), stdout=subprocess.PIPE, shell=True)
+        diff_process.wait() # Make sure svn-create-patch is done before we continue.
+        
+        description = options.description or "patch"
+        tool.bugs.add_patch_to_bug(bug_id, diff_process.stdout, description, mark_for_review=options.review)
+
+
+class PostCommitsAsPatchesToBug(Command):
+    def __init__(self):
+        options = [
+            make_option("--no-review", action="store_false", dest="review", default=True, help="Do not mark the patch for review."),
+        ]
+        Command.__init__(self, 'Attaches a range of local commits to a bug as patch files.', 'BUGID COMMITISH', options=options)
+
+    def execute(self, options, args, tool):
+        bug_id = args[0]
+        
+        if not tool.scm().supports_local_commits():
+            log(tool.scm().display_name() + " does not support local commits.")
+            exit(1)
+        
+        commit_ids = tool.scm().commit_ids_from_range_arguments(args[1:])
+        
+        if len(commit_ids) > 10:
+            log("Are you sure you want to attach %d patches to bug %s?" % (len(commit_ids), bug_id))
+            # Could add a --patches-limit option.
+            exit(1)
+        
+        log("Attaching %d commits as patches to bug %s" % (len(commit_ids), bug_id))
+        for commit_id in commit_ids:
+            commit_message = tool.scm().commit_message_for_commit(commit_id)
+            commit_lines = commit_message.splitlines()
+            
+            description = commit_lines[0]
+            comment_text = "\n".join(commit_lines[1:])
+        
+            comment_text += "\n---\n"
+            comment_text += tool.scm().files_changed_summary_for_commit(commit_id)
+        
+            # This is a little bit of a hack, that we pass stdout as the patch file.
+            # We could alternatively make an in-memory file-like object with the patch contents.
+            diff_process = subprocess.Popen(tool.scm().show_diff_command_for_commit(commit_id), stdout=subprocess.PIPE, shell=True)
+            tool.bugs.add_patch_to_bug(bug_id, diff_process.stdout, description, comment_text, mark_for_review=options.review)
+
+
+class NonWrappingEpilogIndentedHelpFormatter(IndentedHelpFormatter):
+    def __init__(self):
+        IndentedHelpFormatter.__init__(self)
+
+    # The standard IndentedHelpFormatter paragraph-wraps the epilog, killing our custom formatting.
+    def format_epilog(self, epilog):
+        if epilog:
+            return "\n" + epilog + "\n"
+        return ""
+
+class BugzillaTool:
+    def __init__(self):
+        self.cached_scm = None
+        self.bugs = Bugzilla()
+        self.commands = [
+            { 'name' : 'bugs-to-commit', 'object' : BugsInCommitQueue() },
+            { 'name' : 'patches-to-commit', 'object' : PatchesInCommitQueue() },
+            { 'name' : 'reviewed-patches', 'object' : ReviewedPatchesOnBug() },
+            { 'name' : 'apply-patches', 'object' : ApplyPatchesFromBug() },
+            { 'name' : 'land-and-update', 'object' : LandAndUpdateBug() },
+            { 'name' : 'land-patches', 'object' : LandPatchesFromBug() },
+            { 'name' : 'commit-message', 'object' : CommitMessageForCurrentDiff() },
+            { 'name' : 'obsolete-attachments', 'object' : ObsoleteAttachmentsOnBug() },
+            { 'name' : 'post-diff', 'object' : PostDiffAsPatchToBug() },
+            { 'name' : 'post-commits', 'object' : PostCommitsAsPatchesToBug() },
+        ]
+        
+        self.global_option_parser = OptionParser(usage=self.usage_line(), formatter=NonWrappingEpilogIndentedHelpFormatter(), epilog=self.commands_usage())
+        self.global_option_parser.add_option("--dry-run", action="store_true", dest="dryrun", help="do not touch remote servers", default=False)
+    
+    def scm(self):
+        # Lazily initialize SCM to not error-out before command line parsing (or when running non-scm commands).
+        original_cwd = os.path.abspath('.')
+        if not self.cached_scm:
+            self.cached_scm = detect_scm_system(original_cwd)
+        
+        if not self.cached_scm:
+            script_directory = os.path.abspath(sys.path[0])
+            webkit_directory = os.path.abspath(os.path.join(script_directory, "../.."))
+            self.cached_scm = detect_scm_system(webkit_directory)
+            if self.cached_scm:
+                log("The current directory (%s) is not a WebKit checkout, using %s" % (original_cwd, webkit_directory))
+            else:
+                error("FATAL: Failed to determine the SCM system for either %s or %s" % (original_cwd, webkit_directory))
+        
+        return self.cached_scm
+    
+    @staticmethod
+    def usage_line():
+        return "Usage: %prog [options] command [command-options] [command-arguments]"
+    
+    def commands_usage(self):
+        commands_text = "Commands:\n"
+        longest_name_length = 0
+        command_rows = []
+        for command in self.commands:
+            command_object = command['object']
+            command_name_and_args = command_object.name_with_arguments(command['name'])
+            command_rows.append({ 'name-and-args': command_name_and_args, 'object': command_object })
+            longest_name_length = max([longest_name_length, len(command_name_and_args)])
+        
+        # Use our own help formatter so as to indent enough.
+        formatter = IndentedHelpFormatter()
+        formatter.indent()
+        formatter.indent()
+        
+        for row in command_rows:
+            command_object = row['object']
+            commands_text += "  " + row['name-and-args'].ljust(longest_name_length + 3) + command_object.help_text + "\n"
+            commands_text += command_object.option_parser.format_option_help(formatter)
+        return commands_text
+
+    def handle_global_args(self, args):
+        (options, args) = self.global_option_parser.parse_args(args)
+        if len(args):
+            # We'll never hit this because split_args splits at the first arg without a leading '-'
+            self.global_option_parser.error("Extra arguments before command: " + args)
+        
+        if options.dryrun:
+            self.scm().dryrun = True
+            self.bugs.dryrun = True
+    
+    @staticmethod
+    def split_args(args):
+        # Assume the first argument which doesn't start with '-' is the command name.
+        command_index = 0
+        for arg in args:
+            if arg[0] != '-':
+                break
+            command_index += 1
+        else:
+            return (args[:], None, [])
+
+        global_args = args[:command_index]
+        command = args[command_index]
+        command_args = args[command_index + 1:]
+        return (global_args, command, command_args)
+    
+    def command_by_name(self, command_name):
+        for command in self.commands:
+            if command_name == command['name']:
+                return command
+        return None
+    
+    def main(self):
+        (global_args, command_name, args_after_command_name) = self.split_args(sys.argv[1:])
+        
+        # Handle --help, etc:
+        self.handle_global_args(global_args)
+        
+        if not command_name:
+            self.global_option_parser.error("No command specified")
+        
+        command = self.command_by_name(command_name)
+        if not command:
+            self.global_option_parser.error(command_name + " is not a recognized command")
+        
+        command_object = command['object']
+        (command_options, command_args) = command_object.parse_args(args_after_command_name)
+        return command_object.execute(command_options, command_args, self)
+
+
+def main():
+    tool = BugzillaTool()
+    return tool.main()
+
+if __name__ == "__main__":
+    main()
diff --git a/WebKitTools/Scripts/modules/__init__.py b/WebKitTools/Scripts/modules/__init__.py
new file mode 100644 (file)
index 0000000..ef65bee
--- /dev/null
@@ -0,0 +1 @@
+# Required for Python to search this directory for module files
diff --git a/WebKitTools/Scripts/modules/bugzilla.py b/WebKitTools/Scripts/modules/bugzilla.py
new file mode 100644 (file)
index 0000000..fc9bbaf
--- /dev/null
@@ -0,0 +1,283 @@
+# Copyright (c) 2009, Google Inc. All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+# 
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# WebKit's Python module for interacting with Bugzilla
+
+import getpass
+import subprocess
+import sys
+import urllib2
+
+try:
+    from BeautifulSoup import BeautifulSoup
+    from mechanize import Browser
+except ImportError, e:
+    print """
+BeautifulSoup and mechanize are required.
+
+To install:
+sudo easy_install BeautifulSoup mechanize
+
+Or from the web:
+http://www.crummy.com/software/BeautifulSoup/
+http://wwwsearch.sourceforge.net/mechanize/
+"""
+    exit(1)
+
+def log(string):
+    print >> sys.stderr, string
+
+# FIXME: This should not depend on git for config storage
+def read_config(key):
+    # Need a way to read from svn too
+    config_process = subprocess.Popen("git config --get bugzilla." + key, stdout=subprocess.PIPE, shell=True)
+    value = config_process.communicate()[0]
+    return_code = config_process.wait()
+
+    if return_code:
+        return None
+    return value.rstrip('\n')
+
+class Bugzilla:
+    def __init__(self, dryrun=False):
+        self.dryrun = dryrun
+        self.authenticated = False
+        
+        # Defaults (until we support better option parsing):
+        self.bug_server = "https://bugs.webkit.org/"
+        
+        self.browser = Browser()
+        # Ignore bugs.webkit.org/robots.txt until we fix it to allow this script
+        self.browser.set_handle_robots(False)
+
+    # This could eventually be a text file
+    reviewer_usernames_to_full_names = {
+        "abarth" : "Adam Barth",
+        "adele" : "Adele Peterson",
+        "ap" : "Alexey Proskuryakov",
+        "ariya.hidayat" : "Ariya Hidayat",
+        "darin" : "Darin Adler",
+        "ddkilzer" : "David Kilzer",
+        "dglazkov" : "Dimitri Glazkov",
+        "eric" : "Eric Seidel",
+        "fishd" : "Darin Fisher",
+        "gns" : "Gustavo Noronha",
+        "hyatt" : "David Hyatt",
+        "jmalonzo" : "Jan Alonzo",
+        "justin.garcia" : "Justin Garcia",
+        "kevino" : "Kevin Ollivier",
+        "levin" : "David Levin",
+        "mitz" : "Dan Bernstein",
+        "mjs" : "Maciej Stachoviak",
+        "mrowe" : "Mark Rowe",
+        "oliver" : "Oliver Hunt",
+        "sam" : "Sam Weinig",
+        "staikos" : "George Staikos",
+        "timothy" : "Timothy Hatcher",
+        "treat" : "Adam Treat",
+        "xan.lopez" : "Xan Lopez",
+        "zecke" : "Holger Freyther",
+        "zimmermann" : "Nikolas Zimmermann",
+    }
+
+    def full_name_from_bugzilla_name(self, bugzilla_name):
+        if not bugzilla_name in self.reviewer_usernames_to_full_names:
+            raise Exception("ERROR: Unknown reviewer! " + bugzilla_name)
+        return self.reviewer_usernames_to_full_names[bugzilla_name]
+
+    def bug_url_for_bug_id(self, bug_id):
+        bug_base_url = self.bug_server + "show_bug.cgi?id="
+        return "%s%s" % (bug_base_url, bug_id)
+    
+    def attachment_url_for_id(self, attachment_id, action="view"):
+        attachment_base_url = self.bug_server + "attachment.cgi?id="
+        return "%s%s&action=%s" % (attachment_base_url, attachment_id, action)
+
+    def fetch_attachments_from_bug(self, bug_id):
+        bug_url = self.bug_url_for_bug_id(bug_id)
+        log("Fetching: " + bug_url)
+
+        page = urllib2.urlopen(bug_url)
+        soup = BeautifulSoup(page)
+    
+        attachment_table = soup.find('table', {'cellspacing':"0", 'cellpadding':"4", 'border':"1"})
+    
+        attachments = []
+        # Grab a list of non-obsoleted patch files 
+        for attachment_row in attachment_table.findAll('tr'):
+            first_cell = attachment_row.find('td')
+            if not first_cell:
+                continue # This is the header, no cells
+            if first_cell.has_key('colspan'):
+                break # this is the last row
+            
+            attachment = {}
+            attachment['obsolete'] = (attachment_row.has_key('class') and attachment_row['class'] == "bz_obsolete")
+            
+            cells = attachment_row.findAll('td')
+            attachment_link = cells[0].find('a')
+            attachment['url'] = self.bug_server + attachment_link['href'] # urls are relative
+            attachment['id'] = attachment['url'].split('=')[1] # e.g. https://bugs.webkit.org/attachment.cgi?id=31223
+            attachment['name'] = attachment_link.string
+            # attachment['type'] = cells[1]
+            # attachment['date'] = cells[2]
+            # attachment['size'] = cells[3]
+            review_status = cells[4]
+            # action_links = cells[5]
+
+            if str(review_status).find("review+") != -1:
+                reviewer = review_status.contents[0].split(':')[0] # name:\n review+\n
+                reviewer_full_name = self.full_name_from_bugzilla_name(reviewer)
+                attachment['reviewer'] = reviewer_full_name
+
+            attachments.append(attachment)
+        return attachments
+
+    def fetch_reviewed_patches_from_bug(self, bug_id):
+        reviewed_patches = []
+        for attachment in self.fetch_attachments_from_bug(bug_id):
+            if 'reviewer' in attachment and not attachment['obsolete']:
+                reviewed_patches.append(attachment)
+        return reviewed_patches
+
+    def fetch_bug_ids_from_commit_queue(self):
+        # FIXME: We should have an option for restricting the search by email.  Example:
+        # unassigned_only = "&emailassigned_to1=1&emailtype1=substring&email1=unassigned"
+        commit_queue_url = "https://bugs.webkit.org/buglist.cgi?query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&field0-0-0=flagtypes.name&type0-0-0=equals&value0-0-0=review%2B"
+        log("Loading commit queue")
+
+        page = urllib2.urlopen(commit_queue_url)
+        soup = BeautifulSoup(page)
+    
+        bug_ids = []
+        # Grab the cells in the first column (which happens to be the bug ids)
+        for bug_link_cell in soup('td', "first-child"): # tds with the class "first-child"
+            bug_link = bug_link_cell.find("a")
+            bug_ids.append(bug_link.string) # the contents happen to be the bug id
+    
+        return bug_ids
+
+    def fetch_patches_from_commit_queue(self):
+        patches_to_land = []
+        for bug_id in self.fetch_bug_ids_from_commit_queue():
+            patches = self.fetch_reviewed_patches_from_bug(bug_id)
+            patches_to_land += patches
+        return patches_to_land
+
+    def authenticate(self, username=None, password=None):
+        if self.authenticated:
+            return
+        
+        if not username:
+            username = read_config("username")
+            if not username:
+                username = raw_input("Bugzilla login: ")
+        if not password:
+            password = read_config("password")
+            if not password:
+                password = getpass.getpass("Bugzilla password for %s: " % username)
+
+        log("Logging in as %s..." % username)
+        if self.dryrun:
+            self.authenticated = True
+            return
+        self.browser.open(self.bug_server + "/index.cgi?GoAheadAndLogIn=1")
+        self.browser.select_form(name="login")
+        self.browser['Bugzilla_login'] = username
+        self.browser['Bugzilla_password'] = password
+        self.browser.submit()
+
+        # We really should check the result codes and try again as necessary
+        self.authenticated = True
+
+    def add_patch_to_bug(self, bug_id, patch_file_object, description, comment_text=None, mark_for_review=False):
+        self.authenticate()
+        
+        log('Adding patch "%s" to bug %s' % (description, bug_id))
+        if self.dryrun:
+            log(comment_text)
+            return
+        
+        self.browser.open(self.bug_server + "/attachment.cgi?action=enter&bugid=" + bug_id)
+        self.browser.select_form(name="entryform")
+        self.browser['description'] = description
+        self.browser['ispatch'] = ("1",)
+        if comment_text:
+            log(comment_text)
+            self.browser['comment'] = comment_text
+        self.browser['flag_type-1'] = ('?',) if mark_for_review else ('X',)
+        self.browser.add_file(patch_file_object, "text/plain", "bugzilla_requires_a_filename.patch")
+        self.browser.submit()
+
+    def obsolete_attachment(self, attachment_id, comment_text = None):
+        self.authenticate()
+
+        log("Obsoleting attachment: %s" % attachment_id)
+        if self.dryrun:
+            log(comment_text)
+            return
+
+        self.browser.open(self.attachment_url_for_id(attachment_id, 'edit'))
+        self.browser.select_form(nr=0)
+        self.browser.find_control('isobsolete').items[0].selected = True
+        # Also clear any review flag (to remove it from review/commit queues)
+        self.browser.find_control(type='select', nr=0).value = ("X",)
+        if comment_text:
+            log(comment_text)
+            self.browser['comment'] = comment_text
+        self.browser.submit()
+    
+    def post_comment_to_bug(self, bug_id, comment_text):
+        self.authenticate()
+
+        log("Adding comment to bug %s" % bug_id)
+        if self.dryrun:
+            log(comment_text)
+            return
+
+        self.browser.open(self.bug_url_for_bug_id(bug_id))
+        self.browser.select_form(name="changeform")
+        self.browser['comment'] = comment_text
+        self.browser.submit()
+
+    def close_bug_as_fixed(self, bug_id, comment_text=None):
+        self.authenticate()
+
+        log("Closing bug %s as fixed" % bug_id)
+        if self.dryrun:
+            log(comment_text)
+            return
+
+        self.browser.open(self.bug_url_for_bug_id(bug_id))
+        self.browser.select_form(name="changeform")
+        if comment_text:
+            log(comment_text)
+            self.browser['comment'] = comment_text
+        self.browser['knob'] = ['resolve']
+        self.browser['resolution'] = ['FIXED']
+        self.browser.submit()
diff --git a/WebKitTools/Scripts/modules/scm.py b/WebKitTools/Scripts/modules/scm.py
new file mode 100644 (file)
index 0000000..76e7138
--- /dev/null
@@ -0,0 +1,319 @@
+# Copyright (c) 2009, Google Inc. All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+# 
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# Python module for interacting with an SCM system (like SVN or Git)
+
+import os
+import re
+import subprocess
+import sys
+
+def log(string):
+    print >> sys.stderr, string
+
+def error(string):
+    log(string)
+    exit(1)
+
+def detect_scm_system(path):
+    if SVN.in_working_directory(path):
+        return SVN(cwd=path)
+    
+    if Git.in_working_directory(path):
+        return Git(cwd=path)
+    
+    return None
+
+class ScriptError(Exception):
+    pass
+
+class SCM:
+    def __init__(self, cwd, dryrun=False):
+        self.cwd = cwd
+        self.checkout_root = self.find_checkout_root(self.cwd)
+        self.dryrun = dryrun
+
+    @staticmethod
+    def run_command(command, cwd=None):
+        return subprocess.Popen(command, stdout=subprocess.PIPE, shell=True, cwd=cwd).communicate()[0].rstrip()
+
+    def ensure_clean_working_directory(self, force):
+        if not force and not self.working_directory_is_clean():
+            log("Working directory has modifications, pass --force-clean or --no-clean to continue.")
+            os.system(self.status_command())
+            exit(1)
+        
+        log("Cleaning the working directory")
+        self.clean_working_directory()
+    
+    def ensure_no_local_commits(self, force):
+        if not self.supports_local_commits():
+            return
+        commits = self.local_commits()
+        if not len(commits):
+            return
+        if not force:
+            log("Working directory has local commits, pass --force-clean to continue.")
+            exit(1)
+        self.discard_local_commits()
+
+    def apply_patch(self, patch):
+        # It's possible that the patch was not made from the root directory.
+        # We should detect and handle that case.
+        return_code = os.system('curl %s | svn-apply --reviewer "%s"' % (patch['url'], patch['reviewer']))
+        if return_code:
+            raise ScriptError("Patch " + patch['url'] + " failed to download and apply.")
+
+    def run_status_and_extract_filenames(self, status_command, status_regexp):
+        filenames = []
+        for line in os.popen(status_command).readlines():
+            match = re.search(status_regexp, line)
+            if not match:
+                continue
+            # status = match.group('status')
+            filename = match.group('filename')
+            filenames.append(filename)
+        return filenames
+
+    @staticmethod
+    def in_working_directory(path):
+        raise NotImplementedError, "subclasses must implement"
+
+    @staticmethod
+    def find_checkout_root(path):
+        raise NotImplementedError, "subclasses must implement"
+
+    def working_directory_is_clean(self):
+        raise NotImplementedError, "subclasses must implement"
+
+    def clean_working_directory(self):
+        raise NotImplementedError, "subclasses must implement"
+
+    def update_webkit(self):
+        raise NotImplementedError, "subclasses must implement"
+
+    def status_command(self):
+        raise NotImplementedError, "subclasses must implement"
+
+    def changed_files(self):
+        raise NotImplementedError, "subclasses must implement"
+
+    def display_name(self):
+        raise NotImplementedError, "subclasses must implement"
+
+    def create_patch_command(self):
+        raise NotImplementedError, "subclasses must implement"
+
+    def commit_with_message(self, message):
+        raise NotImplementedError, "subclasses must implement"
+    
+    # Subclasses must indicate if they support local commits,
+    # but the SCM baseclass will only call local_commits methods when this is true.
+    def supports_local_commits(self):
+        raise NotImplementedError, "subclasses must implement"
+
+    def discard_local_commits(self):
+        pass
+
+    def local_commits(self):
+        return []
+
+class SVN(SCM):
+    def __init__(self, cwd, dryrun=False):
+        SCM.__init__(self, cwd, dryrun)
+        self.cached_version = None
+    
+    @staticmethod
+    def in_working_directory(path):
+        return os.path.isdir(os.path.join(path, '.svn'))
+    
+    @staticmethod
+    def find_checkout_root(path):
+        last_path = None
+        while True:
+            if not SVN.in_working_directory(path):
+                return last_path
+            last_path = path
+            (path, last_component) = os.path.split(path)
+            if last_path == path:
+                return None
+    
+    def svn_version(self):
+        if not self.cached_version:
+            self.cached_version = self.run_command("svn --version --quiet")
+        
+        return self.cached_version
+
+    def working_directory_is_clean(self):
+        diff_process = subprocess.Popen("svn diff", stdout=subprocess.PIPE, shell=True)
+        diff_output = diff_process.communicate()[0]
+        if diff_process.wait():
+            log("Failed to run svn diff")
+            return False
+        return diff_output == ""
+
+    def clean_working_directory(self):
+        os.system("svn reset -R")
+
+    def update_webkit(self):
+        os.system("update-webkit")
+
+    def status_command(self):
+        return 'svn status'
+
+    def changed_files(self):
+        if self.svn_version() > "1.6":
+            status_regexp = "^(?P<status>[ACDMR]).{6} (?P<filename>.+)$"
+        else:
+            status_regexp = "^(?P<status>[ACDMR]).{5} (?P<filename>.+)$"
+        return self.run_status_and_extract_filenames(self.status_command(), status_regexp)
+
+    def supports_local_commits(self):
+        return False
+
+    def display_name(self):
+        return "svn"
+
+    def create_patch_command(self):
+        return "svn-create-patch"
+
+    def commit_with_message(self, message):
+        commit_process = subprocess.Popen('svn commit -F -', stdin=subprocess.PIPE, shell=True)
+        commit_process.communicate(message)
+
+# All git-specific logic should go here.
+class Git(SCM):
+    def __init__(self, cwd, dryrun=False):
+        SCM.__init__(self, cwd, dryrun)
+    
+    @staticmethod
+    def in_working_directory(path):
+        return SCM.run_command("git rev-parse --is-inside-work-tree 2>&1", cwd=path) == "true"
+
+    @staticmethod
+    def find_checkout_root(path):
+        # "git rev-parse --show-cdup" would be another way to get to the root
+        (checkout_root, dot_git) = os.path.split(SCM.run_command("git rev-parse --git-dir", cwd=path))
+        # If we were using 2.6 # checkout_root = os.path.relpath(checkout_root, path)
+        if not os.path.isabs(checkout_root): # Sometimes git returns relative paths
+            checkout_root = os.path.join(path, checkout_root)
+        return checkout_root
+    
+    def discard_local_commits(self):
+        os.system("git reset --hard trunk")
+    
+    def local_commits(self):
+        return self.run_command("git log --pretty=oneline head...trunk").splitlines()
+    
+    def working_directory_is_clean(self):
+        diff_process = subprocess.Popen("git diff-index head", stdout=subprocess.PIPE, shell=True)
+        diff_output = diff_process.communicate()[0]
+        if diff_process.wait():
+            log("Failed to run git diff-index head")
+            return False
+        return diff_output == ""
+    
+    def clean_working_directory(self):
+        os.system("git reset --hard head")
+        # Could run git clean here too, but that wouldn't match working_directory_is_clean
+    
+    def update_webkit(self):
+        # FIXME: Should probably call update-webkit, no?
+        log("Updating working directory")
+        os.system("git svn rebase")
+
+    def status_command(self):
+        return 'git status'
+
+    def changed_files(self):
+        status_command = 'git diff -r --name-status -C -M'
+        status_regexp = '^(?P<status>[ADM])\t(?P<filename>.+)$'
+        return self.run_status_and_extract_filenames(status_command, status_regexp)
+    
+    def supports_local_commits(self):
+        return True
+
+    def display_name(self):
+        return "git"
+
+    def create_patch_command(self):
+        return "git diff head"
+
+    def commit_with_message(self, message):
+        self.commit_locally_with_message(message)
+        self.push_local_commits_to_server()
+
+    # Git-specific methods:
+    
+    def commit_locally_with_message(self, message):
+        commit_process = subprocess.Popen('git commit -a -F -', stdin=subprocess.PIPE, shell=True)
+        commit_process.communicate(message)
+        
+    def push_local_commits_to_server(self):
+        if self.dryrun:
+            return "Dry run, no remote commit."
+        commit_process = subprocess.Popen('git svn dcommit', stdout=subprocess.PIPE, shell=True)
+        (out, error) = commit_process.communicate()
+        return_code = commit_process.wait()
+        log("Commit failure: " + return_code) # We really should handle the failure
+        return out
+
+    def commit_ids_from_range_arguments(self, args, cherry_pick=False):
+        # First get the commit-ids for the passed in revisions.
+        rev_parse_args = ['git', 'rev-parse', '--revs-only'] + args
+        revisions = self.run_command(" ".join(rev_parse_args)).splitlines()
+        
+        if cherry_pick:
+            return revisions
+        
+        # If we're not cherry picking and were only passed one revision, assume "^revision head" aka "revision..head".
+        if len(revisions) < 2:
+            revisions[0] = "^" + revisions[0]
+            revisions.append("head")
+        
+        rev_list_args = ['git', 'rev-list'] + revisions
+        return self.run_command(" ".join(rev_list_args)).splitlines()
+
+    def commit_message_for_commit(self, commit_id):
+        commit_message_process = subprocess.Popen("git cat-file commit " + commit_id, stdout=subprocess.PIPE, shell=True)
+        commit_message = commit_message_process.communicate()[0]
+        commit_lines = commit_message.splitlines()
+
+        # Skip the git headers.
+        first_line_after_headers = 0
+        for line in commit_lines:
+            first_line_after_headers += 1
+            if line == "":
+                break
+        return "\n".join(commit_lines[first_line_after_headers:])
+
+    def show_diff_command_for_commit(self, commit_id):
+        return "git diff-tree -p " + commit_id
+
+    def files_changed_summary_for_commit(self, commit_id):
+        return subprocess.Popen("git diff-tree --shortstat --no-commit-id " + commit_id, stdout=subprocess.PIPE, shell=True).communicate()[0]