Restructure generate-js-bindings script to be modular and testable
authorbburg@apple.com <bburg@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 22 Oct 2015 04:39:01 +0000 (04:39 +0000)
committerbburg@apple.com <bburg@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 22 Oct 2015 04:39:01 +0000 (04:39 +0000)
commit99c30b3dfd7ccb03f253b9f36ffa9cc4a7057419
treef443d4b9c4b43c2fd8376e5f790c30cb4b1adf5b
parent27bfaf397b8a16694396b4d06854ba7bcc697d9f
Restructure generate-js-bindings script to be modular and testable
https://bugs.webkit.org/show_bug.cgi?id=149929

Reviewed by Alex Christensen.

Source/JavaScriptCore:

This is a new code generator, based on the replay inputs code generator and
the inspector protocol code generator, which produces various files for JS
builtins.

Relative to the generator it replaces, this one consolidates two scripts in
JavaScriptCore and WebCore into a single script with multiple files. Parsed
information about the builtins file is stored in backend-independent model
objects. Each output file has its own code generator that uses the model to
produce resulting code. Generators are additionally parameterized by the target
framework (to choose correct macros and includes) and output mode (one
header/implementation file per builtin or per framework).

It includes a few simple tests of the generator's functionality. These result-
based tests will become increasingly more important as we start to add support
for builtins annotation such as @optional, @internal, etc. to the code generator.

Some of these complexities, such as having two output modes, will be removed in
subsequent patches. This patch is intended to exactly replace the existing
functionality with a unified script that makes additional cleanups straightforward.

Additional cleanup and consolidation between inspector code generator scripts
and this script will be pursued in followup patches.

New tests:

Scripts/tests/builtins/JavaScriptCore-Builtin.Promise-Combined.js
Scripts/tests/builtins/JavaScriptCore-Builtin.Promise-Separate.js
Scripts/tests/builtins/JavaScriptCore-Builtin.prototype-Combined.js
Scripts/tests/builtins/JavaScriptCore-Builtin.prototype-Separate.js
Scripts/tests/builtins/JavaScriptCore-BuiltinConstructor-Combined.js
Scripts/tests/builtins/JavaScriptCore-BuiltinConstructor-Separate.js
Scripts/tests/builtins/WebCore-GuardedBuiltin-Separate.js
Scripts/tests/builtins/WebCore-GuardedInternalBuiltin-Separate.js
Scripts/tests/builtins/WebCore-UnguardedBuiltin-Separate.js
Scripts/tests/builtins/WebCore-xmlCasingTest-Separate.js

* CMakeLists.txt:

    Copy the scripts that are used by other targets to a staging directory inside
    ${DERIVED_SOURCES_DIR}/ForwardingHeaders/JavaScriptCore/Scripts.
    Define JavaScriptCore_SCRIPTS_DIR to point here so that the add_custom_command
    and shared file lists are identical between JavaScriptCore and WebCore. The staged
    scripts are a dependency of the main JavaScriptCore target so that they are
    always staged, even if JavaScriptCore itself does not use a particular script.

    The output files additionally depend on all builtin generator script files
    and input files that are combined into the single header/implementation file.

* DerivedSources.make:

    Define JavaScriptCore_SCRIPTS_DIR explicitly so the rule for code generation and
    shared file lists are identical between JavaScriptCore and WebCore.

    The output files additionally depend on all builtin generator script files
    and input files that are combined into the single header/implementation file.

* JavaScriptCore.xcodeproj/project.pbxproj:

    Mark the new builtins generator files as private headers so we can use them from
    WebCore.

* Scripts/UpdateContents.py: Renamed from Source/JavaScriptCore/UpdateContents.py.
* Scripts/builtins/__init__.py: Added.
* Scripts/builtins/builtins.py: Added.
* Scripts/builtins/builtins_generator.py: Added. This file contains the base generator.
(WK_lcfirst):
(WK_ucfirst):
(BuiltinsGenerator):
(BuiltinsGenerator.__init__):
(BuiltinsGenerator.model):
(BuiltinsGenerator.generate_license):
(BuiltinsGenerator.generate_includes_from_entries):
(BuiltinsGenerator.generate_output):
(BuiltinsGenerator.output_filename):
(BuiltinsGenerator.mangledNameForFunction):
(BuiltinsGenerator.mangledNameForFunction.toCamel):
(BuiltinsGenerator.generate_embedded_code_string_section_for_function):
* Scripts/builtins/builtins_model.py: Added. This file contains builtins model objects.
(ParseException):
(Framework):
(Framework.__init__):
(Framework.setting):
(Framework.fromString):
(Frameworks):
(BuiltinObject):
(BuiltinObject.__init__):
(BuiltinFunction):
(BuiltinFunction.__init__):
(BuiltinFunction.fromString):
(BuiltinFunction.__str__):
(BuiltinsCollection):
(BuiltinsCollection.__init__):
(BuiltinsCollection.parse_builtins_file):
(BuiltinsCollection.copyrights):
(BuiltinsCollection.all_functions):
(BuiltinsCollection._parse_copyright_lines):
(BuiltinsCollection._parse_functions):
* Scripts/builtins/builtins_templates.py: Added.
(BuiltinsGeneratorTemplates):
* Scripts/builtins/builtins_generate_combined_header.py: Added.
(BuiltinsCombinedHeaderGenerator):
(BuiltinsCombinedHeaderGenerator.__init__):
(BuiltinsCombinedHeaderGenerator.output_filename):
(BuiltinsCombinedHeaderGenerator.generate_output):
(BuiltinsCombinedHeaderGenerator.generate_forward_declarations):
(FunctionExecutable):
(VM):
(ConstructAbility):
(generate_section_for_object):
(generate_externs_for_object):
(generate_macros_for_object):
(generate_defines_for_object):
(generate_section_for_code_table_macro):
(generate_section_for_code_name_macro):
* Scripts/builtins/builtins_generate_combined_implementation.py: Added.
(BuiltinsCombinedImplementationGenerator):
(BuiltinsCombinedImplementationGenerator.__init__):
(BuiltinsCombinedImplementationGenerator.output_filename):
(BuiltinsCombinedImplementationGenerator.generate_output):
(BuiltinsCombinedImplementationGenerator.generate_header_includes):
* Scripts/builtins/builtins_generate_separate_header.py: Added.
(BuiltinsSeparateHeaderGenerator):
(BuiltinsSeparateHeaderGenerator.__init__):
(BuiltinsSeparateHeaderGenerator.output_filename):
(BuiltinsSeparateHeaderGenerator.macro_prefix):
(BuiltinsSeparateHeaderGenerator.generate_output):
(BuiltinsSeparateHeaderGenerator.generate_forward_declarations):
(FunctionExecutable):
(generate_header_includes):
(generate_section_for_object):
(generate_externs_for_object):
(generate_macros_for_object):
(generate_defines_for_object):
(generate_section_for_code_table_macro):
(generate_section_for_code_name_macro):
* Scripts/builtins/builtins_generate_separate_implementation.py: Added.
(BuiltinsSeparateImplementationGenerator):
(BuiltinsSeparateImplementationGenerator.__init__):
(BuiltinsSeparateImplementationGenerator.output_filename):
(BuiltinsSeparateImplementationGenerator.macro_prefix):
(BuiltinsSeparateImplementationGenerator.generate_output):
(BuiltinsSeparateImplementationGenerator.generate_header_includes):
* Scripts/builtins/builtins_generate_separate_wrapper.py: Added.
(BuiltinsSeparateWrapperGenerator):
(BuiltinsSeparateWrapperGenerator.__init__):
(BuiltinsSeparateWrapperGenerator.output_filename):
(BuiltinsSeparateWrapperGenerator.macro_prefix):
(BuiltinsSeparateWrapperGenerator.generate_output):
(BuiltinsSeparateWrapperGenerator.generate_header_includes):
* Scripts/generate-js-builtins.py: Added.

    Parse command line options, decide which generators and output modes to use.

(generate_bindings_for_builtins_files):
* Scripts/lazywriter.py: Copied from the inspector protocol generator.
(LazyFileWriter):
(LazyFileWriter.__init__):
(LazyFileWriter.write):
(LazyFileWriter.close):
* Scripts/tests/builtins/JavaScriptCore-Builtin.Promise-Combined.js: Added.
* Scripts/tests/builtins/JavaScriptCore-Builtin.Promise-Separate.js: Added.
* Scripts/tests/builtins/JavaScriptCore-Builtin.prototype-Combined.js: Added.
* Scripts/tests/builtins/JavaScriptCore-Builtin.prototype-Separate.js: Added.
* Scripts/tests/builtins/JavaScriptCore-BuiltinConstructor-Combined.js: Added.
* Scripts/tests/builtins/JavaScriptCore-BuiltinConstructor-Separate.js: Added.
* Scripts/tests/builtins/WebCore-GuardedBuiltin-Separate.js: Added.
* Scripts/tests/builtins/WebCore-GuardedInternalBuiltin-Separate.js: Added.
* Scripts/tests/builtins/WebCore-UnguardedBuiltin-Separate.js: Added.
* Scripts/tests/builtins/WebCore-xmlCasingTest-Separate.js: Added.
* Scripts/tests/builtins/expected/JavaScriptCore-Builtin.Promise-Combined.js-result: Added.
* Scripts/tests/builtins/expected/JavaScriptCore-Builtin.Promise-Separate.js-result: Added.
* Scripts/tests/builtins/expected/JavaScriptCore-Builtin.prototype-Combined.js-result: Added.
* Scripts/tests/builtins/expected/JavaScriptCore-Builtin.prototype-Separate.js-result: Added.
* Scripts/tests/builtins/expected/JavaScriptCore-BuiltinConstructor-Combined.js-result: Added.
* Scripts/tests/builtins/expected/JavaScriptCore-BuiltinConstructor-Separate.js-result: Added.
* Scripts/tests/builtins/expected/WebCore-GuardedBuiltin-Separate.js-result: Added.
* Scripts/tests/builtins/expected/WebCore-GuardedInternalBuiltin-Separate.js-result: Added.
* Scripts/tests/builtins/expected/WebCore-UnguardedBuiltin-Separate.js-result: Added.
* Scripts/tests/builtins/expected/WebCore-xmlCasingTest-Separate.js-result: Added.
* builtins/BuiltinExecutables.cpp:
(JSC::BuiltinExecutables::BuiltinExecutables):
* builtins/BuiltinExecutables.h:
* create_hash_table:

    Update the generated builtin macro names.

* generate-js-builtins: Removed.

Source/WebCore:

* CMakeLists.txt:

    Define JavaScriptCore_SCRIPTS_DIR explicitly so the add_custom_command and
    shared file lists are identical between JavaScriptCore and WebCore.

    The output files additionally depend on all builtin generator script files.

* DerivedSources.make:

    Use JavaScriptCore_SCRIPTS_DIR so that the rule for code generation and
    shared file lists are identical between JavaScriptCore and WebCore.

    The output files additionally depend on all builtin generator script files.

* WebCore.xcodeproj/project.pbxproj:

    Define JavaScriptCore_SCRIPTS_DIR before calling DerivedSources.make.
    This will eventually be merged with the other similar script paths.

* bindings/js/JSDOMWindowBase.cpp:
(WebCore::JSDOMWindowBase::finishCreation):

    Update the generated builtin macro names.

* generate-js-builtins: Removed.

Tools:

Add a stub shell script and basic webkitpy support for running builtins
generator tests.

* Scripts/run-builtins-generator-tests: Added.
(main):
* Scripts/webkitpy/codegen/__init__.py: Added.
* Scripts/webkitpy/codegen/main.py: Added.

    The only interesting difference here from the inspector protocol
    generator equivalent is that this implementation decodes the target
    framework and output mode (combined or separate) from the test's file name.

(BuiltinsGeneratorTests):
(BuiltinsGeneratorTests.__init__):
(BuiltinsGeneratorTests.generate_from_js_builtins):
(BuiltinsGeneratorTests.write_error_file):
(BuiltinsGeneratorTests.detect_changes):
(BuiltinsGeneratorTests.run_tests):
(BuiltinsGeneratorTests.main):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@191433 268f45cc-cd09-0410-ab3c-d52691b4dbfc
53 files changed:
Source/JavaScriptCore/CMakeLists.txt
Source/JavaScriptCore/ChangeLog
Source/JavaScriptCore/DerivedSources.make
Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
Source/JavaScriptCore/PlatformWin.cmake
Source/JavaScriptCore/Scripts/UpdateContents.py [moved from Source/JavaScriptCore/UpdateContents.py with 96% similarity]
Source/JavaScriptCore/Scripts/builtins/__init__.py [new file with mode: 0644]
Source/JavaScriptCore/Scripts/builtins/builtins.py [new file with mode: 0644]
Source/JavaScriptCore/Scripts/builtins/builtins_generate_combined_header.py [new file with mode: 0644]
Source/JavaScriptCore/Scripts/builtins/builtins_generate_combined_implementation.py [new file with mode: 0644]
Source/JavaScriptCore/Scripts/builtins/builtins_generate_separate_header.py [new file with mode: 0644]
Source/JavaScriptCore/Scripts/builtins/builtins_generate_separate_implementation.py [new file with mode: 0644]
Source/JavaScriptCore/Scripts/builtins/builtins_generate_separate_wrapper.py [new file with mode: 0644]
Source/JavaScriptCore/Scripts/builtins/builtins_generator.py [new file with mode: 0644]
Source/JavaScriptCore/Scripts/builtins/builtins_model.py [new file with mode: 0644]
Source/JavaScriptCore/Scripts/builtins/builtins_templates.py [new file with mode: 0644]
Source/JavaScriptCore/Scripts/generate-js-builtins [deleted file]
Source/JavaScriptCore/Scripts/generate-js-builtins.py [new file with mode: 0644]
Source/JavaScriptCore/Scripts/lazywriter.py [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/JavaScriptCore-Builtin.Promise-Combined.js [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/JavaScriptCore-Builtin.Promise-Separate.js [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/JavaScriptCore-Builtin.prototype-Combined.js [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/JavaScriptCore-Builtin.prototype-Separate.js [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/JavaScriptCore-BuiltinConstructor-Combined.js [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/JavaScriptCore-BuiltinConstructor-Separate.js [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/WebCore-GuardedBuiltin-Separate.js [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/WebCore-GuardedInternalBuiltin-Separate.js [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/WebCore-UnguardedBuiltin-Separate.js [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/WebCore-xmlCasingTest-Separate.js [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/expected/JavaScriptCore-Builtin.Promise-Combined.js-result [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/expected/JavaScriptCore-Builtin.Promise-Separate.js-result [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/expected/JavaScriptCore-Builtin.prototype-Combined.js-result [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/expected/JavaScriptCore-Builtin.prototype-Separate.js-result [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/expected/JavaScriptCore-BuiltinConstructor-Combined.js-result [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/expected/JavaScriptCore-BuiltinConstructor-Separate.js-result [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/expected/JavaScriptCore-Operations.Promise-Combined.js-result [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/expected/WebCore-GuardedBuiltin-Separate.js-result [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/expected/WebCore-GuardedInternalBuiltin-Separate.js-result [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/expected/WebCore-UnguardedBuiltin-Separate.js-result [new file with mode: 0644]
Source/JavaScriptCore/Scripts/tests/builtins/expected/WebCore-xmlCasingTest-Separate.js-result [new file with mode: 0644]
Source/JavaScriptCore/builtins/BuiltinExecutables.cpp
Source/JavaScriptCore/builtins/BuiltinExecutables.h
Source/JavaScriptCore/create_hash_table
Source/WebCore/CMakeLists.txt
Source/WebCore/ChangeLog
Source/WebCore/DerivedSources.make
Source/WebCore/WebCore.xcodeproj/project.pbxproj
Source/WebCore/bindings/js/JSDOMWindowBase.cpp
Source/WebCore/generate-js-builtins [deleted file]
Tools/ChangeLog
Tools/Scripts/run-builtins-generator-tests [new file with mode: 0755]
Tools/Scripts/webkitpy/codegen/__init__.py [new file with mode: 0644]
Tools/Scripts/webkitpy/codegen/main.py [new file with mode: 0644]