Add the file uploading capability to the perf dashboard.
authorrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 16 Mar 2017 20:53:35 +0000 (20:53 +0000)
committerrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 16 Mar 2017 20:53:35 +0000 (20:53 +0000)
https://bugs.webkit.org/show_bug.cgi?id=169737

Reviewed by Chris Dumez.

Added /privileged-api/upload-file to upload a file, and /api/uploaded-file/ to download the file
and retrieve its meta data based on its SHA256. We treat two files with the identical SHA256 as
identical since anyone who can upload a file using this mechanism can execute arbitrary code in
our bots anyway. This is important for avoiding uploading a large darwinup roots multiple times
to the server, saving both user's time/bandwidth and server's disk space.

* config.json: Added uploadDirectory, uploadFileLimitInMB, and uploadUserQuotaInMB as options.
* init-database.sql: Added uploaded_files table.

* public/api/uploaded-file.php: Added.
(main): /api/uploaded-file/N would download uploaded_file with id=N. /api/uploaded-file/?sha256=X
would return the meta data for uploaded_file with sha256=X.
(stream_file_content): Streams the file content in 64KB chunks. We support Range & If-Range HTTP
request headers so that browsers can pause and resume downloading of a large root file.
(parse_range_header): Parses Range HTTP request header.

* public/include/json-header.php:
(remote_user_name): Use the default argument of NULL.

* public/include/manifest-generator.php:
(ManifestGenerator::generate): Include the maximum upload size in the manifest file to let the
frontend code preemptively check the file size before attempting to submit a file.

* public/include/uploaded-file-helpers.php: Added.
(format_uploaded_file):
(uploaded_file_path_for_row):

* public/privileged-api/upload-file-form.html: Added. For debugging purposes.
(fetchCSRFfToken):
(upload):

* public/privileged-api/upload-file.php: Added.
(main):
(query_total_file_size):
(create_uploaded_file_from_form_data):

* public/shared/common-remote.js:
(CommonRemoteAPI.prototype.postFormData): Added.
(CommonRemoteAPI.prototype.postFormDataWithStatus): Added.
(CommonRemoteAPI.prototype.sendHttpRequestWithFormData): Added.
(CommonRemoteAPI.prototype._asJSON): Throw an exception instead of calling a non-existent reject.

* public/v3/models/uploaded-file.js: Added.
(UploadedFile): Added.
(UploadedFile.uploadFile): Added.
(UploadedFile.fetchUnloadedFileWithIdenticalHash): Added. Finds the file with the same SHA256 in
the server to avoid uploading a large custom root multiple times.
(UploadedFile._computeSHA256Hash): Added.

* public/v3/privileged-api.js:
(PrivilegedAPI.prototype.sendRequest): Added the options dictionary as a third argument. For now,
only support useFormData boolean.

* public/v3/remote.js:
(BrowserRemoteAPI.prototype.sendHttpRequestWithFormData): Added.

* server-tests/api-manifest.js: Updated per the inclusion of fileUploadSizeLimit in the manifest.
* server-tests/api-uploaded-file.js: Added.
* server-tests/privileged-api-upload-file-tests.js: Added.

* server-tests/resources/temporary-file.js: Added.
(TemporaryFile): Added. A helper class for creating a temporary file to upload.
(TemporaryFile.makeTemporaryFileOfSizeInMB):
(TemporaryFile.makeTemporaryFile):
(TemporaryFile.inject):

* server-tests/resources/test-server.conf: Set upload_max_filesize and post_max_size for testing.
* server-tests/resources/test-server.js:
(TestServer.prototype.testConfig): Use uploadFileLimitInMB and uploadUserQuotaInMB of 2MB and 5MB.
(TestServer.prototype._ensureDataDirectory): Create a directory to store uploaded files inside
the data directory. In a production server, we can place it outside ServerRoot / DocumentRoot.
(TestServer.prototype.cleanDataDirectory): Delete the aforementioned directory as needed.

* tools/js/database.js:
(tableToPrefixMap): Added uploaded_files.

* tools/js/remote.js:
(NodeRemoteAPI.prototype.sendHttpRequest): Added a dictionary to specify request headers and
a callback to process the response as arguments. Fixed the bug that any 2xx code other than 200
was resulting in a rejected promise. Also include the response headers in the result for tests.
Finally, when content is a function, call that instead of writing the content since FormData
requires a custom logic.
(NodeRemoteAPI.prototype.sendHttpRequestWithFormData): Added.

* tools/js/v3-models.js: Include uploaded-file.js.

* tools/run-tests.py:
(main): Add form-data as a new dependency.

git-svn-id: http://svn.webkit.org/repository/webkit/trunk@214065 268f45cc-cd09-0410-ab3c-d52691b4dbfc

23 files changed:
Websites/perf.webkit.org/ChangeLog
Websites/perf.webkit.org/config.json
Websites/perf.webkit.org/init-database.sql
Websites/perf.webkit.org/public/api/uploaded-file.php [new file with mode: 0644]
Websites/perf.webkit.org/public/include/json-header.php
Websites/perf.webkit.org/public/include/manifest-generator.php
Websites/perf.webkit.org/public/include/uploaded-file-helpers.php [new file with mode: 0644]
Websites/perf.webkit.org/public/privileged-api/upload-file-form.html [new file with mode: 0644]
Websites/perf.webkit.org/public/privileged-api/upload-file.php [new file with mode: 0644]
Websites/perf.webkit.org/public/shared/common-remote.js
Websites/perf.webkit.org/public/v3/models/uploaded-file.js [new file with mode: 0644]
Websites/perf.webkit.org/public/v3/privileged-api.js
Websites/perf.webkit.org/public/v3/remote.js
Websites/perf.webkit.org/server-tests/api-manifest.js
Websites/perf.webkit.org/server-tests/api-uploaded-file.js [new file with mode: 0644]
Websites/perf.webkit.org/server-tests/privileged-api-upload-file-tests.js [new file with mode: 0644]
Websites/perf.webkit.org/server-tests/resources/temporary-file.js [new file with mode: 0644]
Websites/perf.webkit.org/server-tests/resources/test-server.conf
Websites/perf.webkit.org/server-tests/resources/test-server.js
Websites/perf.webkit.org/tools/js/database.js
Websites/perf.webkit.org/tools/js/remote.js
Websites/perf.webkit.org/tools/js/v3-models.js
Websites/perf.webkit.org/tools/run-tests.py

index 82ed260..fdba6fd 100644 (file)
@@ -1,3 +1,99 @@
+2017-03-16  Ryosuke Niwa  <rniwa@webkit.org>
+
+        Add the file uploading capability to the perf dashboard.
+        https://bugs.webkit.org/show_bug.cgi?id=169737
+
+        Reviewed by Chris Dumez.
+
+        Added /privileged-api/upload-file to upload a file, and /api/uploaded-file/ to download the file
+        and retrieve its meta data based on its SHA256. We treat two files with the identical SHA256 as
+        identical since anyone who can upload a file using this mechanism can execute arbitrary code in
+        our bots anyway. This is important for avoiding uploading a large darwinup roots multiple times
+        to the server, saving both user's time/bandwidth and server's disk space.
+
+        * config.json: Added uploadDirectory, uploadFileLimitInMB, and uploadUserQuotaInMB as options.
+        * init-database.sql: Added uploaded_files table.
+
+        * public/api/uploaded-file.php: Added.
+        (main): /api/uploaded-file/N would download uploaded_file with id=N. /api/uploaded-file/?sha256=X
+        would return the meta data for uploaded_file with sha256=X.
+        (stream_file_content): Streams the file content in 64KB chunks. We support Range & If-Range HTTP
+        request headers so that browsers can pause and resume downloading of a large root file.
+        (parse_range_header): Parses Range HTTP request header.
+
+        * public/include/json-header.php:
+        (remote_user_name): Use the default argument of NULL.
+
+        * public/include/manifest-generator.php:
+        (ManifestGenerator::generate): Include the maximum upload size in the manifest file to let the
+        frontend code preemptively check the file size before attempting to submit a file.
+
+        * public/include/uploaded-file-helpers.php: Added.
+        (format_uploaded_file):
+        (uploaded_file_path_for_row):
+
+        * public/privileged-api/upload-file-form.html: Added. For debugging purposes.
+        (fetchCSRFfToken):
+        (upload):
+
+        * public/privileged-api/upload-file.php: Added.
+        (main):
+        (query_total_file_size):
+        (create_uploaded_file_from_form_data):
+
+        * public/shared/common-remote.js:
+        (CommonRemoteAPI.prototype.postFormData): Added.
+        (CommonRemoteAPI.prototype.postFormDataWithStatus): Added.
+        (CommonRemoteAPI.prototype.sendHttpRequestWithFormData): Added.
+        (CommonRemoteAPI.prototype._asJSON): Throw an exception instead of calling a non-existent reject.
+
+        * public/v3/models/uploaded-file.js: Added.
+        (UploadedFile): Added.
+        (UploadedFile.uploadFile): Added.
+        (UploadedFile.fetchUnloadedFileWithIdenticalHash): Added. Finds the file with the same SHA256 in
+        the server to avoid uploading a large custom root multiple times.
+        (UploadedFile._computeSHA256Hash): Added.
+
+        * public/v3/privileged-api.js:
+        (PrivilegedAPI.prototype.sendRequest): Added the options dictionary as a third argument. For now,
+        only support useFormData boolean.
+
+        * public/v3/remote.js:
+        (BrowserRemoteAPI.prototype.sendHttpRequestWithFormData): Added.
+
+        * server-tests/api-manifest.js: Updated per the inclusion of fileUploadSizeLimit in the manifest.
+        * server-tests/api-uploaded-file.js: Added.
+        * server-tests/privileged-api-upload-file-tests.js: Added.
+
+        * server-tests/resources/temporary-file.js: Added.
+        (TemporaryFile): Added. A helper class for creating a temporary file to upload.
+        (TemporaryFile.makeTemporaryFileOfSizeInMB):
+        (TemporaryFile.makeTemporaryFile):
+        (TemporaryFile.inject):
+
+        * server-tests/resources/test-server.conf: Set upload_max_filesize and post_max_size for testing.
+        * server-tests/resources/test-server.js:
+        (TestServer.prototype.testConfig): Use uploadFileLimitInMB and uploadUserQuotaInMB of 2MB and 5MB.
+        (TestServer.prototype._ensureDataDirectory): Create a directory to store uploaded files inside
+        the data directory. In a production server, we can place it outside ServerRoot / DocumentRoot.
+        (TestServer.prototype.cleanDataDirectory): Delete the aforementioned directory as needed.
+
+        * tools/js/database.js:
+        (tableToPrefixMap): Added uploaded_files.
+
+        * tools/js/remote.js:
+        (NodeRemoteAPI.prototype.sendHttpRequest): Added a dictionary to specify request headers and
+        a callback to process the response as arguments. Fixed the bug that any 2xx code other than 200
+        was resulting in a rejected promise. Also include the response headers in the result for tests.
+        Finally, when content is a function, call that instead of writing the content since FormData
+        requires a custom logic.
+        (NodeRemoteAPI.prototype.sendHttpRequestWithFormData): Added.
+
+        * tools/js/v3-models.js: Include uploaded-file.js.
+
+        * tools/run-tests.py:
+        (main): Add form-data as a new dependency.
+
 2017-03-15  Dewei Zhu  <dewei_zhu@apple.com>
 
         Fix unit test and bug fix for 'pull-os-versions.js' script.
index bd35457..e6d99a2 100644 (file)
@@ -3,6 +3,9 @@
     "debug": true,
     "jsonCacheMaxAge": 600,
     "dataDirectory": "public/data/",
+    "uploadDirectory": "uploaded",
+    "uploadFileLimitInMB": 800,
+    "uploadUserQuotaInMB": 8192,
     "database": {
         "host": "localhost",
         "port": "5432",
index 5c6bb94..77e3f0c 100644 (file)
@@ -24,6 +24,7 @@ DROP TYPE IF EXISTS analysis_task_result_type CASCADE;
 DROP TABLE IF EXISTS build_triggerables CASCADE;
 DROP TABLE IF EXISTS triggerable_configurations CASCADE;
 DROP TABLE IF EXISTS triggerable_repositories CASCADE;
+DROP TABLE IF EXISTS uploaded_files CASCADE;
 DROP TABLE IF EXISTS bugs CASCADE;
 DROP TABLE IF EXISTS analysis_test_groups CASCADE;
 DROP TABLE IF EXISTS commit_sets CASCADE;
@@ -241,6 +242,19 @@ CREATE TABLE triggerable_configurations (
     trigconfig_triggerable integer REFERENCES build_triggerables NOT NULL,
     CONSTRAINT triggerable_must_be_unique_for_test_and_platform UNIQUE(trigconfig_test, trigconfig_platform));
 
+CREATE TABLE uploaded_files (
+    file_id serial PRIMARY KEY,
+    file_created_at timestamp NOT NULL DEFAULT (CURRENT_TIMESTAMP AT TIME ZONE 'UTC'),
+    file_deleted_at timestamp,
+    file_mime varchar(64),
+    file_filename varchar(1024) NOT NULL,
+    file_extension varchar(16),
+    file_author varchar(256),
+    file_size bigint NOT NULL,
+    file_sha256 char(64) NOT NULL);
+CREATE INDEX file_author_index ON uploaded_files(file_author);
+CREATE UNIQUE INDEX file_sha256_index ON uploaded_files(file_sha256) WHERE file_deleted_at is NULL;
+
 CREATE TABLE analysis_test_groups (
     testgroup_id serial PRIMARY KEY,
     testgroup_task integer REFERENCES analysis_tasks NOT NULL,
diff --git a/Websites/perf.webkit.org/public/api/uploaded-file.php b/Websites/perf.webkit.org/public/api/uploaded-file.php
new file mode 100644 (file)
index 0000000..0d639b5
--- /dev/null
@@ -0,0 +1,122 @@
+<?php
+
+require('../include/json-header.php');
+require('../include/uploaded-file-helpers.php');
+
+function main($path)
+{
+    if (count($path) > 1)
+        exit_with_error('InvalidRequest');
+
+    $db = connect();
+    if (count($path) && $path[0]) {
+        $file_id = intval($path[0]);
+        $file_row = $db->select_first_row('uploaded_files', 'file', array('id' => $file_id));
+        if (!$file_row)
+            exit_with_404();
+        $file_path = uploaded_file_path_for_row($file_row);
+        return stream_file_content($file_path, $file_row['file_sha256'], $file_row['file_filename']);
+    }
+
+    $sha256 = array_get($_GET, 'sha256');
+    if ($sha256) {
+        $file_row = $db->select_first_row('uploaded_files', 'file', array('sha256' => $sha256, 'deleted_at' => null));
+        if (!$file_row)
+            exit_with_error('NotFound');
+        exit_with_success(array('uploadedFile' => format_uploaded_file($file_row)));
+    }
+    exit_with_error('InvalidArguments');
+}
+
+define('STREAM_CHUNK_SIZE', 64 * 1024);
+
+function stream_file_content($uploaded_file, $etag, $disposition_name)
+{
+    if (!file_exists($uploaded_file))
+        exit_with_404();
+
+    $file_size = filesize($uploaded_file);
+    $last_modified = gmdate('D, d M Y H:i:s', filemtime($uploaded_file)) . ' GMT';
+    $file_handle = fopen($uploaded_file, "rb");
+    if (!$file_handle)
+        exit_with_404();
+
+    $headers = getallheaders();
+
+    // We don't support multi-part range request. e.g. bytes=1-3,4-5
+    $range = parse_range_header(array_get($headers, 'Range'), $file_size);
+    if ($range && (!array_key_exists('If-Range', $headers) || $headers['If-Range'] == $last_modified || $headers['If-Range'] == $etag)) {
+        assert($range['start'] >= 0);
+        if ($range['start'] > $range['end'] || $range['end'] >= $file_size) {
+            header('HTTP/1.1 416 Range Not Satisfiable');
+            header("Content-Range: bytes */$file_size");
+            exit(416);
+        }
+        $start = $range['start'];
+        $end = $range['end'];
+        $content_length = $end - $start + 1;
+        header('HTTP/1.1 206 Partial Content');
+        header("Content-Range: bytes $start-$end/$file_size");
+        fseek($file_handle, $start);
+    } else {
+        $content_length = $file_size;
+        header("Accept-Ranges: bytes");
+        header("ETag: $etag");
+    }
+
+    $output_buffer = fopen('php://output', 'wb');
+    $encoded_filename = urlencode($disposition_name);
+
+    header('Content-Type: application/octet-stream');
+    header('Content-Length: ' . $content_length);
+    header("Content-Disposition: attachment; filename*=utf-8''$encoded_filename");
+    header("Last-Modified: $last_modified");
+
+    set_time_limit(0);
+    while (!feof($file_handle) && $content_length) {
+        $is_end = $content_length < STREAM_CHUNK_SIZE;
+        $chunk_size = $is_end ? $content_length : STREAM_CHUNK_SIZE;
+        $chunk = fread($file_handle, $chunk_size);
+        $content_length -= $chunk_size;
+        fwrite($output_buffer, $chunk, $chunk_size);
+        flush();
+    }
+
+    exit(0);
+}
+
+function parse_range_header($range_header, $file_size)
+{
+    // We don't support multi-part range request. e.g. bytes=1-3,4-5
+    $matches = array();
+    $end_byte = $file_size;
+    if (!$range_header || !preg_match('/^\s*bytes\s*=\s*((\d+)-(\d*)|-(\d+))\s*$/', $range_header, $matches))
+        return NULL;
+
+    $end_byte = $file_size - 1;
+    if ($matches[2]) {
+        $start_byte = intval($matches[2]);
+        if ($matches[3])
+            $end_byte = intval($matches[3]);
+        else
+            $end_byte = $file_size - 1;
+    } else {
+        $suffix_length = intval($matches[4]);
+        if ($file_size < $suffix_length)
+            $start_byte = 0;
+        else
+            $start_byte = $file_size - $suffix_length;
+    }
+
+    return array('start' => $start_byte, 'end' => $end_byte);
+}
+
+function exit_with_404()
+{
+    header($_SERVER['SERVER_PROTOCOL'] . ' 404 Not Found');
+    exit_with_error('NotFound');
+}
+
+main(array_key_exists('PATH_INFO', $_SERVER) ? explode('/', trim($_SERVER['PATH_INFO'], '/')) : array());
+
+?>
index 415117d..c1c8dca 100644 (file)
@@ -1,6 +1,7 @@
 <?php
 
 require_once('db.php');
+
 require_once('test-path-resolver.php');
 
 header('Content-type: application/json');
@@ -116,8 +117,8 @@ function ensure_privileged_api_data_and_token() {
     return $data;
 }
 
-function remote_user_name($data) {
-    return should_authenticate_as_slave($data) ? NULL : array_get($_SERVER, 'REMOTE_USER');
+function remote_user_name($data = NULL) {
+    return $data && should_authenticate_as_slave($data) ? NULL : array_get($_SERVER, 'REMOTE_USER');
 }
 
 function should_authenticate_as_slave($data) {
index 9ec772c..ec63373 100644 (file)
@@ -43,6 +43,7 @@ class ManifestGenerator {
             'triggerables'=> (object)$this->triggerables(),
             'dashboards' => (object)config('dashboards'),
             'summaryPages' => config('summaryPages'),
+            'fileUploadSizeLimit' => config('uploadFileLimitInMB', 0) * 1024 * 1024,
         );
 
         $this->manifest['elapsedTime'] = (microtime(true) - $start_time) * 1000;
diff --git a/Websites/perf.webkit.org/public/include/uploaded-file-helpers.php b/Websites/perf.webkit.org/public/include/uploaded-file-helpers.php
new file mode 100644 (file)
index 0000000..51b6c57
--- /dev/null
@@ -0,0 +1,20 @@
+<?php
+
+function format_uploaded_file($file_row)
+{
+    return array(
+        'id' => $file_row['file_id'],
+        'size' => $file_row['file_size'],
+        'createdAt' => $file_row['file_created_at'],
+        'mime' => $file_row['file_mime'],
+        'filename' => $file_row['file_filename'],
+        'author' => $file_row['file_author'],
+        'sha256' => $file_row['file_sha256']);
+}
+
+function uploaded_file_path_for_row($file_row)
+{
+    return config_path('uploadDirectory', $file_row['file_id'] . $file_row['file_extension']);
+}
+
+?>
diff --git a/Websites/perf.webkit.org/public/privileged-api/upload-file-form.html b/Websites/perf.webkit.org/public/privileged-api/upload-file-form.html
new file mode 100644 (file)
index 0000000..275cd85
--- /dev/null
@@ -0,0 +1,59 @@
+<!DOCTYPE html>
+<html>
+<body>
+<script>
+
+function fetchCSRFfToken(fileInput, progressElement) {
+    return new Promise((resolve, reject) => {
+        const xhr = new XMLHttpRequest();
+        xhr.open('POST', 'generate-csrf-token', true);
+        xhr.onload = () => {
+            let content;
+            try {
+                content = JSON.parse(xhr.responseText);
+            } catch (error) {
+                return reject(error + ':' + xhr.responseText);
+            }
+            if (content['status'] != 'OK')
+                reject(content['status']);
+            else
+                resolve(content['token']);
+        }
+        xhr.onerror = reject;
+        xhr.send('{}');
+    });
+}
+
+function upload(fileInput, progressElement) {
+    fetchCSRFfToken().then((token) => {
+        const xhr = new XMLHttpRequest();
+        const formData = new FormData();
+        formData.append('token', token);
+        formData.append('newFile', fileInput.files[0]);
+
+        xhr.open('POST', 'upload-file', true);
+        xhr.onload = function () {
+            alert(xhr.response);
+        }
+        xhr.onerror = function () {
+            alert('error: ' + xhr.response);
+        }
+        xhr.upload.onprogress = function (event) {
+            if (event.lengthComputable) {
+                progressElement.max = event.total;
+                progressElement.value = event.loaded;
+            }
+        }
+        xhr.send(formData);
+    }, (error) => {
+        alert(`Failed to fetch the CSRF token: ${error}`);
+    });
+}
+
+</script>
+<p>Upload a new custom root. <b>This is for debuging purpose only. The uploaded file will be deleted.</b></p>
+<input id=file type="file">
+<button onclick="upload(file, p)">Upload</button>
+<progress id=p></progress>
+</body>
+</html>
diff --git a/Websites/perf.webkit.org/public/privileged-api/upload-file.php b/Websites/perf.webkit.org/public/privileged-api/upload-file.php
new file mode 100644 (file)
index 0000000..bb65db2
--- /dev/null
@@ -0,0 +1,99 @@
+<?php
+
+ini_set('upload_max_filesize', '1025M');
+ini_set('post_max_size', '1025M');
+require_once('../include/json-header.php');
+require_once('../include/uploaded-file-helpers.php');
+
+define('MEGABYTES', 1024 * 1024);
+
+function main()
+{
+    if (array_get($_SERVER, 'CONTENT_LENGTH') && empty($_POST) && empty($_FILES))
+        exit_with_error('FileSizeLimitExceeded2');
+
+    if (!verify_token(array_get($_POST, 'token')))
+        exit_with_error('InvalidToken');
+
+    if (!is_dir(config_path('uploadDirectory', '')))
+        exit_with_error('NotSupported');
+
+    $input_file = array_get($_FILES, 'newFile');
+    if (!$input_file)
+        exit_with_error('NoFileSpecified');
+
+    if ($input_file['error'] != UPLOAD_ERR_OK)
+        exit_with_error('FailedToUploadFile', array('name' => $input_file['name'], 'error' => $input_file['error']));
+
+    if (config('uploadFileLimitInMB') * MEGABYTES < $input_file['size'])
+        exit_with_error('FileSizeLimitExceeded');
+
+    $uploaded_file = create_uploaded_file_from_form_data($input_file);
+
+    $current_user = remote_user_name();
+    $db = connect();
+
+    // FIXME: Cleanup old files.
+
+    if (config('uploadUserQuotaInMB') * MEGABYTES - query_total_file_size($db, $current_user) < $input_file['size'])
+        exit_with_error('FileSizeQuotaExceeded');
+
+    $db->begin_transaction();
+    $file_row = $db->select_or_insert_row('uploaded_files', 'file',
+        array('sha256' => $uploaded_file['sha256'], 'deleted_at' => null), $uploaded_file, '*');
+    if (!$file_row)
+        exit_with_error('FailedToInsertFileData');
+
+    // A concurrent session may have inserted another file.
+    if (config('uploadUserQuotaInMB') * MEGABYTES < query_total_file_size($db, $current_user)) {
+        $db->rollback_transaction();
+        exit_with_error('FileSizeQuotaExceeded');
+    }
+
+    $new_path = uploaded_file_path_for_row($file_row);
+    if (!move_uploaded_file($input_file['tmp_name'], $new_path)) {
+        $db->rollback_transaction();
+        exit_with_error('FailedToMoveUploadedFile');
+    }
+    $db->commit_transaction();
+
+    exit_with_success(array('uploadedFile' => format_uploaded_file($file_row)));
+}
+
+function query_total_file_size($db, $user)
+{
+    if ($user)
+        $count_result = $db->query_and_fetch_all('SELECT sum(file_size) as "sum" FROM uploaded_files WHERE file_deleted_at IS NULL AND file_author = $1', array($user));
+    else
+        $count_result = $db->query_and_fetch_all('SELECT sum(file_size) as "sum" FROM uploaded_files WHERE file_deleted_at IS NULL AND file_author IS NULL');
+    if (!$count_result)
+        return FALSE;
+    return intval($count_result[0]["sum"]);
+}
+
+function create_uploaded_file_from_form_data($input_file)
+{
+    $file_sha256 = hash_file('sha256', $input_file['tmp_name']);
+    if (!$file_sha256)
+        exit_with_error('FailedToComputeSHA256');
+
+    $matches = array();
+    $file_extension = null;
+    if (preg_match('/(\.[a-zA-Z0-9]{1,5}){1,2}$/', $input_file['name'], $matches)) {
+        $file_extension = $matches[0];
+        assert(strlen($file_extension) <= 16);
+    }
+
+    return array(
+        'author' => remote_user_name(),
+        'filename' => $input_file['name'],
+        'extension' => $file_extension,
+        'mime' => $input_file['type'], // Sanitize MIME types.
+        'size' => $input_file['size'],
+        'sha256' => $file_sha256
+    );
+}
+
+main();
+
+?>
index a7f3024..e7aa85e 100644 (file)
@@ -11,6 +11,19 @@ class CommonRemoteAPI {
         return this._checkStatus(this.postJSON(path, data));
     }
 
+    postFormData(path, data)
+    {
+        const formData = new FormData();
+        for (let key in data)
+            formData.append(key, data[key]);
+        return this._asJSON(this.sendHttpRequestWithFormData(path, formData));
+    }
+
+    postFormDataWithStatus(path, data)
+    {
+        return this._checkStatus(this.postFormData(path, data));
+    }
+
     getJSON(path)
     {
         return this._asJSON(this.sendHttpRequest(path, 'GET', null, null));
@@ -26,6 +39,11 @@ class CommonRemoteAPI {
         throw 'NotImplemented';
     }
 
+    sendHttpRequestWithFormData(path, formData)
+    {
+        throw 'NotImplemented';
+    }
+
     _asJSON(promise)
     {
         return promise.then((result) => {
@@ -33,7 +51,7 @@ class CommonRemoteAPI {
                 return JSON.parse(result.responseText);
             } catch (error) {
                 console.error(result.responseText);
-                reject(result.statusCode + ', ' + error);
+                throw `{result.statusCode}: ${error}`;
             }
         });
     }
diff --git a/Websites/perf.webkit.org/public/v3/models/uploaded-file.js b/Websites/perf.webkit.org/public/v3/models/uploaded-file.js
new file mode 100644 (file)
index 0000000..f401384
--- /dev/null
@@ -0,0 +1,54 @@
+
+class UploadedFile extends DataModelObject {
+
+    constructor(id, object)
+    {
+        super(id, object);
+        this._createdAt = new Date(object.createdAt);
+        this._filename = object.filename;
+        this._author = object.author;
+        this._size = object.size;
+        this._sha256 = object.sha256;
+        this.ensureNamedStaticMap('sha256')[object.sha256] = this;
+    }
+
+    static uploadFile(file)
+    {
+        return PrivilegedAPI.sendRequest('upload-file', {'newFile': file}, {useFormData: true}).then((rawData) => {
+            return UploadedFile.ensureSingleton(rawData['uploadedFile'].id, rawData['uploadedFile']);
+        });
+    }
+
+    static fetchUnloadedFileWithIdenticalHash(file)
+    {
+        return new Promise((resolve, reject) => {
+            const reader = new FileReader();
+            reader.onload = () => resolve(reader.result);
+            reader.onerror = () => reject();
+            reader.readAsArrayBuffer(file);
+        }).then((content) => {
+            return this._computeSHA256Hash(content);
+        }).then((sha256) => {
+            const map = this.namedStaticMap('sha256');
+            if (map && sha256 in map)
+                return map[sha256];
+            return RemoteAPI.getJSONWithStatus(`../api/uploaded-file?sha256=${sha256}`).then((rawData) => {
+                if (!rawData['uploadedFile'])
+                    return null;
+                return UploadedFile.ensureSingleton(rawData['uploadedFile'].id, rawData['uploadedFile']);
+            });
+        });
+    }
+
+    static _computeSHA256Hash(content)
+    {
+        return crypto.subtle.digest('SHA-256', content).then((digest) => {
+            return Array.from(new Uint8Array(digest)).map((byte) => {
+                if (byte < 0x10)
+                    return '0' + byte.toString(16);
+                return byte.toString(16);
+            }).join('');
+        });
+    }
+
+}
index a4ca38f..267f94b 100644 (file)
@@ -2,14 +2,16 @@
 
 class PrivilegedAPI {
 
-    static sendRequest(path, data)
+    static sendRequest(path, data, options = {useFormData: false})
     {
         const clonedData = {};
         for (let key in data)
             clonedData[key] = data[key];
 
         const fullPath = '/privileged-api/' + path;
-        const post = () => RemoteAPI.postJSONWithStatus(fullPath, clonedData);
+        const post = options.useFormData
+            ? () => RemoteAPI.postFormDataWithStatus(fullPath, clonedData)
+            : () => RemoteAPI.postJSONWithStatus(fullPath, clonedData);
 
         return this.requestCSRFToken().then((token) => {
             clonedData['token'] = token;
index bfdef03..0eee833 100644 (file)
@@ -36,6 +36,11 @@ class BrowserRemoteAPI extends CommonRemoteAPI {
         });
     }
 
+    sendHttpRequestWithFormData(path, formData)
+    {
+        return this.sendHttpRequest(path, 'POST', null, formData); // Content-type is set by the browser.
+    }
+
 }
 
 const RemoteAPI = new BrowserRemoteAPI;
index cea851a..833835c 100644 (file)
@@ -14,7 +14,7 @@ describe('/api/manifest', function () {
     it("should generate an empty manifest when database is empty", () => {
         return TestServer.remoteAPI().getJSON('/api/manifest').then((manifest) => {
             assert.deepEqual(Object.keys(manifest).sort(), ['all', 'bugTrackers', 'builders', 'dashboard', 'dashboards',
-                'elapsedTime', 'metrics', 'repositories', 'siteTitle', 'status', 'summaryPages', 'tests', 'triggerables']);
+                'elapsedTime', 'fileUploadSizeLimit', 'metrics', 'repositories', 'siteTitle', 'status', 'summaryPages', 'tests', 'triggerables']);
 
             assert.equal(typeof(manifest.elapsedTime), 'number');
             delete manifest.elapsedTime;
@@ -26,6 +26,7 @@ describe('/api/manifest', function () {
                 builders: {},
                 dashboard: {},
                 dashboards: {},
+                fileUploadSizeLimit: 2097152, // 2MB during testing.
                 metrics: {},
                 repositories: {},
                 tests: {},
diff --git a/Websites/perf.webkit.org/server-tests/api-uploaded-file.js b/Websites/perf.webkit.org/server-tests/api-uploaded-file.js
new file mode 100644 (file)
index 0000000..5ea1e9d
--- /dev/null
@@ -0,0 +1,309 @@
+'use strict';
+
+require('../tools/js/v3-models.js');
+
+const assert = require('assert');
+global.FormData = require('form-data');
+
+const TestServer = require('./resources/test-server.js');
+const TemporaryFile = require('./resources/temporary-file.js').TemporaryFile;
+
+describe('/api/uploaded-file', function () {
+    this.timeout(5000);
+    TestServer.inject();
+
+    TemporaryFile.inject();
+
+    it('should return "InvalidArguments" when neither path nor sha256 query is set', () => {
+        return TestServer.remoteAPI().getJSON('/api/uploaded-file').then((content) => {
+            assert.equal(content['status'], 'InvalidArguments');
+            return TestServer.remoteAPI().getJSON('/api/uploaded-file/');
+        }).then((content) => {
+            assert.equal(content['status'], 'InvalidArguments');
+        });
+    });
+
+    it('should return 404 when there is no file with the specified ID', () => {
+        return TestServer.remoteAPI().getJSON('/api/uploaded-file/1').then((content) => {
+            assert(false, 'should never be reached');
+        }, (error) => {
+            assert.equal(error, 404);
+        });
+    });
+
+    it('should return 404 when the specified ID is not a valid integer', () => {
+        return TestServer.remoteAPI().getJSON('/api/uploaded-file/foo').then((content) => {
+            assert(false, 'should never be reached');
+        }, (error) => {
+            assert.equal(error, 404);
+        });
+    });
+
+    it('should return the file content matching the specified file ID', () => {
+        let uploadedFile;
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            uploadedFile = response['uploadedFile'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${uploadedFile['id']}`, 'GET', null, null);
+        }).then((response) => {
+            assert.equal(response.responseText, 'some content');
+        });
+    });
+
+    it('should return "NotFound" when the specified SHA256 is invalid', () => {
+        return TestServer.remoteAPI().getJSON('/api/uploaded-file/?sha256=abc').then((content) => {
+            assert.equal(content['status'], 'NotFound');
+        });
+    });
+
+    it('should return "NotFound" when there is no file matching the specified SHA256 ', () => {
+        return TestServer.remoteAPI().getJSON('/api/uploaded-file/?sha256=5256ec18f11624025905d057d6befb03d77b243511ac5f77ed5e0221ce6d84b5').then((content) => {
+            assert.equal(content['status'], 'NotFound');
+        });
+    });
+
+    it('should return the meta data of the file with the specified SHA256', () => {
+        let uploadedFile;
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            uploadedFile = response['uploadedFile'];
+            return TestServer.remoteAPI().getJSON(`/api/uploaded-file/?sha256=${uploadedFile['sha256']}`);
+        }).then((response) => {
+            assert.deepEqual(uploadedFile, response['uploadedFile']);
+        });
+    });
+
+    it('should return "NotFound" when the file matching the specified SHA256 had already been deleted', () => {
+        let uploadedFile;
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            uploadedFile = response['uploadedFile'];
+            const db = TestServer.database();
+            return db.connect().then(() => db.query(`UPDATE uploaded_files SET file_deleted_at = now() at time zone 'utc'`));
+        }).then(() => {
+            return TestServer.remoteAPI().getJSON(`/api/uploaded-file/?sha256=${uploadedFile['sha256']}`);
+        }).then((content) => {
+            assert.equal(content['status'], 'NotFound');
+        });
+    });
+
+
+    it('should respond with ETag, Acccept-Ranges, Content-Disposition, Content-Length, and Last-Modified headers', () => {
+        let uploadedFile;
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            uploadedFile = response['uploadedFile'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${uploadedFile['id']}`, 'GET', null, null);
+        }).then((response) => {
+            const headers = response.headers;
+
+            assert(Object.keys(headers).includes('etag'));
+            assert.equal(headers['etag'], uploadedFile['sha256']);
+
+            assert(Object.keys(headers).includes('accept-ranges'));
+            assert.equal(headers['accept-ranges'], 'bytes');
+
+            assert(Object.keys(headers).includes('content-disposition'));
+            assert.equal(headers['content-disposition'], `attachment; filename*=utf-8''some.dat`);
+
+            assert(Object.keys(headers).includes('content-length'));
+            assert.equal(headers['content-length'], uploadedFile['size']);
+
+            assert(Object.keys(headers).includes('last-modified'));
+        });
+    });
+
+    it('should respond with the same Last-Modified each time', () => {
+        let id;
+        let lastModified;
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            id = response['uploadedFile']['id'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null);
+        }).then((response) => {
+            lastModified = response.headers['last-modified'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null);
+        }).then((response) => {
+            assert.equal(response.headers['last-modified'], lastModified);
+        });
+    });
+
+    it('should respond with Content-Range when requested after X bytes', () => {
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            const id = response['uploadedFile']['id'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null, {'Range': 'bytes=5-'});
+        }).then((response) => {
+            const headers = response.headers;
+            assert.equal(response.statusCode, 206);
+            assert.equal(headers['content-range'], 'bytes 5-11/12');
+            assert.equal(response.responseText, 'content');
+        });
+    });
+
+    it('should respond with Content-Range when requested between X-Y bytes', () => {
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            const id = response['uploadedFile']['id'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null, {'Range': 'bytes=4-9'});
+        }).then((response) => {
+            const headers = response.headers;
+            assert.equal(response.statusCode, 206);
+            assert.equal(headers['content-range'], 'bytes 4-9/12');
+            assert.equal(response.responseText, ' conte');
+        });
+    });
+
+    it('should respond with Content-Range when requested for the last X bytes', () => {
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            const id = response['uploadedFile']['id'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null, {'Range': 'bytes=-4'});
+        }).then((response) => {
+            const headers = response.headers;
+            assert.equal(response.statusCode, 206);
+            assert.equal(headers['content-range'], 'bytes 8-11/12');
+            assert.equal(response.responseText, 'tent');
+        });
+    });
+
+    it('should respond with Content-Range for the whole content when the suffix length is larger than the content', () => {
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            const id = response['uploadedFile']['id'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null, {'Range': 'bytes=-100'});
+        }).then((response) => {
+            const headers = response.headers;
+            assert.equal(response.statusCode, 206);
+            assert.equal(headers['content-range'], 'bytes 0-11/12');
+            assert.equal(response.responseText, 'some content');
+        });
+    });
+
+    it('should return 416 when the starting byte is after the file size', () => {
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            const id = response['uploadedFile']['id'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null, {'Range': 'bytes=12-'})
+                .then(() => assert(false, 'should never be reached'), (error) => assert.equal(error, 416));
+        });
+    });
+
+    it('should return 416 when the starting byte after the ending byte', () => {
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            const id = response['uploadedFile']['id'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null, {'Range': 'bytes=2-1'})
+                .then(() => assert(false, 'should never be reached'), (error) => assert.equal(error, 416));
+        });
+    });
+
+    it('should respond with Content-Range when If-Range matches the last modified date', () => {
+        let id;
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            id = response['uploadedFile']['id'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null);
+        }).then((response) => {
+            assert.equal(response.statusCode, 200);
+            assert.equal(response.responseText, 'some content');
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null,
+                {'Range': 'bytes = 9-10', 'If-Range': response.headers['last-modified']});
+        }).then((response) => {
+            const headers = response.headers;
+            assert.equal(response.statusCode, 206);
+            assert.equal(headers['content-range'], 'bytes 9-10/12');
+            assert.equal(response.responseText, 'en');
+        });
+    });
+
+    it('should respond with Content-Range when If-Range matches ETag', () => {
+        let id;
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            id = response['uploadedFile']['id'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null);
+        }).then((response) => {
+            assert.equal(response.statusCode, 200);
+            assert.equal(response.responseText, 'some content');
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null,
+                {'Range': 'bytes = 9-10', 'If-Range': response.headers['etag']});
+        }).then((response) => {
+            const headers = response.headers;
+            assert.equal(response.statusCode, 206);
+            assert.equal(headers['content-range'], 'bytes 9-10/12');
+            assert.equal(response.responseText, 'en');
+        });
+    });
+
+    it('should return the full content when If-Range does not match the last modified date or ETag', () => {
+        let id;
+        return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            id = response['uploadedFile']['id'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null);
+        }).then((response) => {
+            assert.equal(response.statusCode, 200);
+            assert.equal(response.responseText, 'some content');
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null,
+                {'Range': 'bytes = 9-10', 'If-Range': 'foo'});
+        }).then((response) => {
+            assert.equal(response.statusCode, 200);
+            assert.equal(response.responseText, 'some content');
+        });
+    });
+
+    it('should respond with Content-Range across 64KB streaming chunks', () => {
+        let id;
+        const fileSize = 256 * 1024;
+        const tokens = "0123456789abcdefghijklmnopqrstuvwxyz";
+        let buffer = Buffer.allocUnsafe(fileSize);
+        for (let i = 0; i < fileSize; i++)
+            buffer[i] = Math.floor(Math.random() * 256);
+        let startByte = 63 * 1024;
+        let endByte = 128 * 1024 - 1;
+
+        let responseBufferList = [];
+        const responseHandler = (response) => {
+            response.on('data', (chunk) => responseBufferList.push(chunk));
+        };
+
+        function verifyBuffer()
+        {
+            const responseBuffer = Buffer.concat(responseBufferList);
+            for (let i = 0; i < endByte - startByte + 1; i++) {
+                const actual = responseBuffer[i];
+                const expected = buffer[startByte + i];
+                assert.equal(actual, expected, `The byte at index ${i} should be identical. Expected ${expected} but got ${actual}`);
+            }
+        }
+
+        return TemporaryFile.makeTemporaryFile('some.dat', buffer).then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            id = response['uploadedFile']['id'];
+            return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null,
+                {'Range': `bytes = ${startByte}-${endByte}`}, responseHandler);
+        }).then((response) => {
+            const headers = response.headers;
+            assert.equal(response.statusCode, 206);
+            assert.equal(headers['content-range'], `bytes ${startByte}-${endByte}/${fileSize}`);
+            verifyBuffer();
+        });
+    });
+
+});
diff --git a/Websites/perf.webkit.org/server-tests/privileged-api-upload-file-tests.js b/Websites/perf.webkit.org/server-tests/privileged-api-upload-file-tests.js
new file mode 100644 (file)
index 0000000..6927785
--- /dev/null
@@ -0,0 +1,192 @@
+'use strict';
+
+require('../tools/js/v3-models.js');
+
+const assert = require('assert');
+global.FormData = require('form-data');
+
+const TestServer = require('./resources/test-server.js');
+const TemporaryFile = require('./resources/temporary-file.js').TemporaryFile;
+
+describe('/privileged-api/upload-file', function () {
+    this.timeout(5000);
+    TestServer.inject();
+
+    TemporaryFile.inject();
+
+    it('should return "NotFileSpecified" when newFile not is specified', () => {
+        return PrivilegedAPI.sendRequest('upload-file', {}, {useFormData: true}).then(() => {
+            assert(false, 'should never be reached');
+        }, (error) => {
+            assert.equal(error, 'NoFileSpecified');
+        });
+    });
+
+    it('should return "FileSizeLimitExceeded" when the file is too big', () => {
+        return TemporaryFile.makeTemporaryFileOfSizeInMB('some.dat', TestServer.testConfig().uploadFileLimitInMB + 1).then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true}).then(() => {
+                assert(false, 'should never be reached');
+            }, (error) => {
+                assert.equal(error, 'FileSizeLimitExceeded');
+            });
+        });
+    });
+
+    it('should upload a file when the filesize is smaller than the limit', () => {
+        const db = TestServer.database();
+        const limitInMB = TestServer.testConfig().uploadFileLimitInMB;
+        let uploadedFile;
+        return TemporaryFile.makeTemporaryFileOfSizeInMB('some.dat', limitInMB).then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            uploadedFile = response['uploadedFile'];
+            return db.connect().then(() => db.selectAll('uploaded_files', 'id'));
+        }).then((rows) => {
+            assert.equal(rows.length, 1);
+            assert.equal(rows[0].id, uploadedFile.id);
+            assert.equal(rows[0].size, limitInMB * 1024 * 1024);
+            assert.equal(rows[0].size, uploadedFile.size);
+            assert.equal(rows[0].filename, 'some.dat');
+            assert.equal(rows[0].filename, uploadedFile.filename);
+            assert.equal(rows[0].extension, '.dat');
+            assert.equal(rows[0].sha256, '5256ec18f11624025905d057d6befb03d77b243511ac5f77ed5e0221ce6d84b5');
+            assert.equal(rows[0].sha256, uploadedFile.sha256);
+        });
+    });
+
+    it('should not create a duplicate files when the identical files are uploaded', () => {
+        const db = TestServer.database();
+        const limitInMB = TestServer.testConfig().uploadFileLimitInMB;
+        let uploadedFile1;
+        let uploadedFile2;
+        return TemporaryFile.makeTemporaryFileOfSizeInMB('some.dat', limitInMB).then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            uploadedFile1 = response['uploadedFile'];
+            return TemporaryFile.makeTemporaryFileOfSizeInMB('other.dat', limitInMB);
+        }).then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            uploadedFile2 = response['uploadedFile'];
+            return db.connect().then(() => db.selectAll('uploaded_files', 'id'));
+        }).then((rows) => {
+            assert.deepEqual(uploadedFile1, uploadedFile2);
+            assert.equal(rows.length, 1);
+            assert.equal(rows[0].id, uploadedFile1.id);
+            assert.equal(rows[0].size, limitInMB * 1024 * 1024);
+            assert.equal(rows[0].size, uploadedFile1.size);
+            assert.equal(rows[0].filename, 'some.dat');
+            assert.equal(rows[0].filename, uploadedFile1.filename);
+            assert.equal(rows[0].extension, '.dat');
+            assert.equal(rows[0].sha256, '5256ec18f11624025905d057d6befb03d77b243511ac5f77ed5e0221ce6d84b5');
+            assert.equal(rows[0].sha256, uploadedFile1.sha256);
+        });
+    });
+
+    it('should not create a duplicate files when the identical files are uploaded', () => {
+        const db = TestServer.database();
+        const limitInMB = TestServer.testConfig().uploadFileLimitInMB;
+        let uploadedFile1;
+        let uploadedFile2;
+        return TemporaryFile.makeTemporaryFileOfSizeInMB('some.dat', limitInMB).then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            uploadedFile1 = response['uploadedFile'];
+            return TemporaryFile.makeTemporaryFileOfSizeInMB('other.dat', limitInMB);
+        }).then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            uploadedFile2 = response['uploadedFile'];
+            return db.connect().then(() => db.selectAll('uploaded_files', 'id'));
+        }).then((rows) => {
+            assert.deepEqual(uploadedFile1, uploadedFile2);
+            assert.equal(rows.length, 1);
+            assert.equal(rows[0].id, uploadedFile1.id);
+            assert.equal(rows[0].size, limitInMB * 1024 * 1024);
+            assert.equal(rows[0].size, uploadedFile1.size);
+            assert.equal(rows[0].filename, 'some.dat');
+            assert.equal(rows[0].filename, uploadedFile1.filename);
+            assert.equal(rows[0].extension, '.dat');
+            assert.equal(rows[0].sha256, '5256ec18f11624025905d057d6befb03d77b243511ac5f77ed5e0221ce6d84b5');
+            assert.equal(rows[0].sha256, uploadedFile1.sha256);
+        });
+    });
+
+    it('should re-upload the file when the previously uploaded file had been deleted', () => {
+        const db = TestServer.database();
+        const limitInMB = TestServer.testConfig().uploadFileLimitInMB;
+        let uploadedFile1;
+        let uploadedFile2;
+        return TemporaryFile.makeTemporaryFileOfSizeInMB('some.dat', limitInMB).then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            uploadedFile1 = response['uploadedFile'];
+            return db.connect().then(() => db.query(`UPDATE uploaded_files SET file_deleted_at = now() at time zone 'utc'`));
+        }).then(() => {
+            return TemporaryFile.makeTemporaryFileOfSizeInMB('other.dat', limitInMB);
+        }).then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then((response) => {
+            uploadedFile2 = response['uploadedFile'];
+            return db.connect().then(() => db.selectAll('uploaded_files', 'id'));
+        }).then((rows) => {
+            assert.notEqual(uploadedFile1.id, uploadedFile2.id);
+            assert.equal(rows.length, 2);
+            assert.equal(rows[0].id, uploadedFile1.id);
+            assert.equal(rows[1].id, uploadedFile2.id);
+
+            assert.equal(rows[0].filename, 'some.dat');
+            assert.equal(rows[0].filename, uploadedFile1.filename);
+            assert.equal(rows[1].filename, 'other.dat');
+            assert.equal(rows[1].filename, uploadedFile2.filename);
+
+            assert.equal(rows[0].size, limitInMB * 1024 * 1024);
+            assert.equal(rows[0].size, uploadedFile1.size);
+            assert.equal(rows[0].size, uploadedFile2.size);
+            assert.equal(rows[0].size, rows[1].size);
+            assert.equal(rows[0].sha256, '5256ec18f11624025905d057d6befb03d77b243511ac5f77ed5e0221ce6d84b5');
+            assert.equal(rows[0].sha256, uploadedFile1.sha256);
+            assert.equal(rows[0].sha256, uploadedFile2.sha256);
+            assert.equal(rows[0].sha256, rows[1].sha256);
+            assert.equal(rows[0].extension, '.dat');
+            assert.equal(rows[1].extension, '.dat');
+        });
+    });
+
+    it('should pick up at most two file extensions', () => {
+        const db = TestServer.database();
+        const limitInMB = TestServer.testConfig().uploadFileLimitInMB;
+        return TemporaryFile.makeTemporaryFileOfSizeInMB('some.other.tar.gz', limitInMB).then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then(() => {
+            return db.connect().then(() => db.selectAll('uploaded_files', 'id'))
+        }).then((rows) => {
+            assert.equal(rows.length, 1);
+            assert.equal(rows[0].size, limitInMB * 1024 * 1024);
+            assert.equal(rows[0].mime, 'application/octet-stream');
+            assert.equal(rows[0].filename, 'some.other.tar.gz');
+            assert.equal(rows[0].extension, '.tar.gz');
+            assert.equal(rows[0].sha256, '5256ec18f11624025905d057d6befb03d77b243511ac5f77ed5e0221ce6d84b5');
+        });
+    });
+
+    it('should return "FileSizeQuotaExceeded" when the total file size exceeds the quota allowed per user', () => {
+        const db = TestServer.database();
+        const limitInMB = TestServer.testConfig().uploadFileLimitInMB;
+        return TemporaryFile.makeTemporaryFileOfSizeInMB('some.dat', limitInMB, 'a').then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then(() => {
+            return TemporaryFile.makeTemporaryFileOfSizeInMB('other.dat', limitInMB, 'b');
+        }).then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+        }).then(() => {
+            return TemporaryFile.makeTemporaryFileOfSizeInMB('other.dat', limitInMB, 'c');
+        }).then((stream) => {
+            return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true}).then(() => {
+                assert(false, 'should never be reached');
+            }, (error) => {
+                assert.equal(error, 'FileSizeQuotaExceeded');
+            });
+        });
+    });
+});
diff --git a/Websites/perf.webkit.org/server-tests/resources/temporary-file.js b/Websites/perf.webkit.org/server-tests/resources/temporary-file.js
new file mode 100644 (file)
index 0000000..7b84b24
--- /dev/null
@@ -0,0 +1,49 @@
+
+const assert = require('assert');
+const childProcess = require('child_process');
+const fs = require('fs');
+const path = require('path');
+
+const Config = require('../../tools/js/config.js');
+
+class TemporaryFile {
+    static makeTemporaryFileOfSizeInMB(name, sizeInMB, characterToFill = 'a')
+    {
+        let megabyteString = characterToFill;
+        for (let i = 0; i < 20; i++)
+            megabyteString = megabyteString + megabyteString;
+        assert.equal(megabyteString.length, 1024 * 1024);
+
+        let content = '';
+        for (let i = 0; i < sizeInMB; i++)
+            content += megabyteString;
+        
+        return this.makeTemporaryFile(name, content);
+    }
+
+    static makeTemporaryFile(name, content)
+    {
+        const newPath = path.resolve(TemporaryFile._tempDir, name);
+        return new Promise((resolve) => {
+            return fs.writeFile(newPath, content, () => {
+                resolve(fs.createReadStream(newPath));
+            });
+        });
+    }
+
+    static inject()
+    {
+        beforeEach(() => {
+            this._tempDir = fs.mkdtempSync(path.resolve(Config.path('dataDirectory'), 'temp/'));
+        });
+
+        afterEach(() => {
+            childProcess.execFileSync('rm', ['-rf', this._tempDir]);
+            this._tempDir = null;
+        });
+    }
+}
+TemporaryFile._tempDir = null;
+
+if (typeof module != 'undefined')
+    module.exports.TemporaryFile = TemporaryFile;
index 59fc8f4..7ac2973 100644 (file)
@@ -52,6 +52,9 @@ LogLevel warn
 <IfModule php5_module>
     AddType application/x-httpd-php .php
     AddType application/x-httpd-php-source .phps
+
+    php_value upload_max_filesize 5M
+    php_value post_max_size 5M
 </IfModule>
 
 Include /private/etc/apache2/extra/httpd-mpm.conf
index 4c50a00..6c3a175 100644 (file)
@@ -76,6 +76,9 @@ class TestServer {
                 'password': Config.value('database.password'),
                 'name': Config.value('testDatabaseName'),
             },
+            'uploadFileLimitInMB': 2,
+            'uploadUserQuotaInMB': 5,
+            'uploadDirectory': Config.value('dataDirectory') + '/uploaded',
             'universalSlavePassword': null,
             'maintenanceMode': false,
             'clusterStart': [2000, 1, 1, 0, 0],
@@ -96,6 +99,7 @@ class TestServer {
         } else if (fs.existsSync(backupPath)) // Assume this is a backup from the last failed run
             this._backupDataPath = backupPath;
         fs.mkdirSync(this._dataDirectory, 0o755);
+        fs.mkdirSync(path.resolve(this._dataDirectory, 'uploaded'), 0o755);
     }
 
     _restoreDataDirectory()
@@ -108,8 +112,13 @@ class TestServer {
     cleanDataDirectory()
     {
         let fileList = fs.readdirSync(this._dataDirectory);
+        for (let filename of fileList) {
+            if (filename != 'uploaded')
+                fs.unlinkSync(path.resolve(this._dataDirectory, filename));
+        }
+        fileList = fs.readdirSync(path.resolve(this._dataDirectory, 'uploaded'));
         for (let filename of fileList)
-            fs.unlinkSync(path.resolve(this._dataDirectory, filename));
+            fs.unlinkSync(path.resolve(this._dataDirectory, 'uploaded', filename));
     }
 
     _ensureTestDatabase()
index 8a80bee..872f5a5 100644 (file)
@@ -152,6 +152,7 @@ const tableToPrefixMap = {
     'commit_sets': 'commitset',
     'commit_set_relationships': 'commitset',
     'run_iterations': 'iteration',
+    'uploaded_files': 'file',
 }
 
 if (typeof module != 'undefined')
index 2ee38b4..9d9fb99 100644 (file)
@@ -64,7 +64,7 @@ class NodeRemoteAPI extends CommonRemoteAPI {
         });
     }
 
-    sendHttpRequest(path, method, contentType, content)
+    sendHttpRequest(path, method, contentType, content, headers = {}, responseHandler = null)
     {
         let server = this._server;
         return new Promise((resolve, reject) => {
@@ -78,10 +78,14 @@ class NodeRemoteAPI extends CommonRemoteAPI {
 
             let request = (server.scheme == 'http' ? http : https).request(options, (response) => {
                 let responseText = '';
-                response.setEncoding('utf8');
-                response.on('data', (chunk) => { responseText += chunk; });
+                if (responseHandler)
+                    responseHandler(response);
+                else {
+                    response.setEncoding('utf8');
+                    response.on('data', (chunk) => { responseText += chunk; });
+                }
                 response.on('end', () => {
-                    if (response.statusCode != 200)
+                    if (response.statusCode < 200 || response.statusCode >= 300)
                         return reject(response.statusCode);
 
                     if ('set-cookie' in response.headers) {
@@ -90,7 +94,7 @@ class NodeRemoteAPI extends CommonRemoteAPI {
                             this._cookies.set(nameValue[0], nameValue[1]);
                         }
                     }
-                    resolve({statusCode: response.statusCode, responseText: responseText});
+                    resolve({statusCode: response.statusCode, responseText: responseText, headers: response.headers});
                 });
             });
 
@@ -102,12 +106,26 @@ class NodeRemoteAPI extends CommonRemoteAPI {
             if (this._cookies.size)
                 request.setHeader('Cookie', Array.from(this._cookies.keys()).map((key) => `${key}=${this._cookies.get(key)}`).join('; '));
 
-            if (content)
-                request.write(content);
+            for (let headerName in headers)
+                request.setHeader(headerName, headers[headerName]);
 
-            request.end();
+            if (content instanceof Function)
+                content(request);
+            else {
+                if (content)
+                    request.write(content);
+                request.end();
+            }
         });
     }
+
+    sendHttpRequestWithFormData(path, formData)
+    {
+        return this.sendHttpRequest(path, 'POST', `multipart/form-data; boundary=${formData.getBoundary()}`, (request) => {
+            formData.pipe(request);
+        });
+    }
+
 };
 
 if (typeof module != 'undefined')
index 7d6e499..31bfd16 100644 (file)
@@ -28,6 +28,7 @@ importFromV3('models/test.js', 'Test');
 importFromV3('models/test-group.js', 'TestGroup');
 importFromV3('models/time-series.js', 'TimeSeries');
 importFromV3('models/triggerable.js', 'Triggerable');
+importFromV3('models/uploaded-file.js', 'UploadedFile');
 
 importFromV3('privileged-api.js', 'PrivilegedAPI');
 importFromV3('instrumentation.js', 'Instrumentation');
index 62f4dac..0398f25 100755 (executable)
@@ -11,7 +11,7 @@ def main():
     node_modules_dir = os.path.join(root_dir, 'node_modules')
 
     os.chdir(root_dir)
-    packages = ['mocha', 'pg']
+    packages = ['mocha', 'pg', 'form-data']
     for package_name in packages:
         target_dir = os.path.join(node_modules_dir, package_name)
         if not os.path.isdir(target_dir):