Commit c9848a2b authored by Joe Hoyle's avatar Joe Hoyle
Browse files

Merge branch 'master' into aws-sdk-3-0

parents 7281a6ff 8cf788b9
Loading
Loading
Loading
Loading
+2 −0
Original line number Diff line number Diff line
.DS_Store
.idea/
*.phar
 No newline at end of file
+73 −11
Original line number Diff line number Diff line
S3-Uploads
==========

[![Build Status](https://travis-ci.org/humanmade/S3-Uploads.svg?branch=master)](https://travis-ci.org/humanmade/S3-Uploads)
[![codecov.io](http://codecov.io/github/humanmade/S3-Uploads/coverage.svg?branch=master)](http://codecov.io/github/humanmade/S3-Uploads?branch=master)

WordPress plugin to store uploads on S3. S3-Uploads aims to be a lightweight "drop-in" for storing uploads on Amazon S3 instead of the local filesystem.
<table width="100%">
	<tr>
		<td align="left" width="70">
			<strong>S3 Uploads</strong><br />
			Lightweight "drop-in" for storing WordPress uploads on Amazon S3 instead of the local filesystem.
		</td>
		<td align="right" width="20%">
			<a href="https://travis-ci.org/humanmade/S3-Uploads">
				<img src="https://travis-ci.org/humanmade/S3-Uploads.svg?branch=master" alt="Build status">
			</a>
			<a href="http://codecov.io/github/humanmade/S3-Uploads?branch=master">
				<img src="http://codecov.io/github/humanmade/S3-Uploads/coverage.svg?branch=master" alt="Coverage via codecov.io" />
			</a>
		</td>
	</tr>
	<tr>
		<td>
			A <strong><a href="https://hmn.md/">Human Made</a></strong> project. Maintained by @joehoyle.
		</td>
		<td align="center">
			<img src="https://hmn.md/content/themes/hmnmd/assets/images/hm-logo.svg" width="100" />
		</td>
	</tr>
</table>

S3 is a WordPress plugin to store uploads on S3. S3-Uploads aims to be a lightweight "drop-in" for storing uploads on Amazon S3 instead of the local filesystem.

It's focused on providing a highly robust S3 interface with no "bells and whistles", WP-Admin UI or much otherwise. It comes with some helpful WP-CLI commands for generating IAM users, listing files on S3 and Migrating your existing library to S3.

@@ -18,6 +37,13 @@ Once you have `git clone`d the repo, or added it as a Git Submodule, add the fol
define( 'S3_UPLOADS_BUCKET', 'my-bucket' );
define( 'S3_UPLOADS_KEY', '' );
define( 'S3_UPLOADS_SECRET', '' );
define( 'S3_UPLOADS_REGION', '' ); // the s3 bucket region, required for Frankfurt and Beijing.
```

You must then enable the plugin. To do this via WP-CLI use command:

```
wp plugin activate S3-Uploads
```

The next thing that you should do is to verify your setup. You can do this using the `verify` command
@@ -75,7 +101,7 @@ There is also an all purpose `cp` command for arbitrary copying to and from S3.
wp s3-uploads cp <from> <to>
```

Note: as either `<from>` or `<to>` can be S3 or local locations, you must speficy the full S3 location via `s3://mybucket/mydirectory` for example `cp ./test.txt s3://mybucket/test.txt`.
Note: as either `<from>` or `<to>` can be S3 or local locations, you must specify the full S3 location via `s3://mybucket/mydirectory` for example `cp ./test.txt s3://mybucket/test.txt`.

Cache Control
==========
@@ -84,15 +110,51 @@ You can define the default HTTP `Cache-Control` header for uploaded media using
following constant:

```PHP
define( 'S3_UPLOADS_CACHE_CONTROL', 30 * 24 * 60 * 60 );
define( 'S3_UPLOADS_HTTP_CACHE_CONTROL', 30 * 24 * 60 * 60 );
	// will expire in 30 days time
```

You can also configure the `Expires` header using the `S3_UPLOADS_EXPIRES` constant
You can also configure the `Expires` header using the `S3_UPLOADS_HTTP_EXPIRES` constant
For instance if you wanted to set an asset to effectively not expire, you could
set the Expires header way off in the future.  For example:

```PHP
define( 'S3_UPLOADS_EXPIRES', gmdate( 'D, d M Y H:i:s', time() + (10 * 365 * 24 * 60 * 60) ) .' GMT' );
define( 'S3_UPLOADS_HTTP_EXPIRES', gmdate( 'D, d M Y H:i:s', time() + (10 * 365 * 24 * 60 * 60) ) .' GMT' );
	// will expire in 10 years time
```

Default Behaviour
==========

As S3 Uploads is a plug and play plugin, activating it will start rewriting image URLs to S3, and also put
new uploads on S3. Sometimes this isn't required behaviour as a site owner may want to upload a large
amount of media to S3 using the `wp-cli` commands before enabling S3 Uploads to direct all uploads requests
to S3. In this case one can define the `S3_UPLOADS_AUTOENABLE` to `false`. For example, place the following
in your `wp-config.php`:

```PHP
define( 'S3_UPLOADS_AUTOENABLE', false );
```

To then enabled S3 Uploads rewriting, use the wp-cli command: `wp s3-uploads enable` / `wp s3-uploads disable`
to toggle the behaviour.

Offline Development
=======

While it's possible to use S3 Uploads for local development (this is actually a nice way to not have to sync all uploads from production to development),
if you want to develop offline you have a couple of options.

1. Just disable the S3 Uploads plugin in your development environment.
2. Define the `S3_UPLOADS_USE_LOCAL` constant with the plugin active.

Option 2 will allow you to run the S3 Uploads plugin for production parity purposes, it will essentially mock
Amazon S3 with a local stream wrapper and actually store the uploads in your WP Upload Dir `/s3/`.

Credits
=======
Created by Human Made for high volume and large-scale sites. We run S3 Uploads on sites with millions of monthly page views, and thousands of sites.

Written and maintained by [Joe Hoyle](https://github.com/joehoyle). Thanks to all our [contributors](https://github.com/humanmade/S3-Uploads/graphs/contributors).

Interested in joining in on the fun? [Join us, and become human!](https://hmn.md/is/hiring/)
+5 −2
Original line number Diff line number Diff line
@@ -10,8 +10,9 @@ class S3_Uploads_Image_Editor_Imagick extends WP_Image_Editor_Imagick {
	protected function _save( $image, $filename = null, $mime_type = null ) {
		list( $filename, $extension, $mime_type ) = $this->get_output_format( $filename, $mime_type );

		if ( ! $filename )
		if ( ! $filename ) {
			$filename = $this->generate_filename( null, null, $extension );
		}

		$upload_dir = wp_upload_dir();

@@ -22,12 +23,14 @@ class S3_Uploads_Image_Editor_Imagick extends WP_Image_Editor_Imagick {
		$save = parent::_save( $image, $temp_filename, $mime_type );

		if ( is_wp_error( $save ) ) {
			unlink( $temp_filename );
			return $save;
		}

		$copy_result = copy( $save['path'], $filename );

		unlink( $save['path'] );
		unlink( $temp_filename );

		if ( ! $copy_result ) {
			return new WP_Error( 'unable-to-copy-to-s3', 'Unable to copy the temp image to S3' );
+76 −80
Original line number Diff line number Diff line
@@ -18,7 +18,7 @@ class S3_Uploads_Local_Stream_Wrapper {
	 *
	 * @var resource
	 */
	public $handle = NULL;
	public $handle = null;

	/**
	 * Instance URI (stream).
@@ -64,7 +64,7 @@ class S3_Uploads_Local_Stream_Wrapper {
	 *   Returns a string representing a location suitable for writing of a file,
	 *   or FALSE if unable to write to the file such as with read-only streams.
	 */
	protected function getTarget($uri = NULL) {
	protected function getTarget( $uri = null ) {
		if ( ! isset( $uri ) ) {
			$uri = $this->uri;
		}
@@ -75,7 +75,7 @@ class S3_Uploads_Local_Stream_Wrapper {
		return trim( $target, '\/' );
	}

	static function getMimeType($uri, $mapping = NULL) {
	static function getMimeType( $uri, $mapping = null ) {

		$extension = '';
		$file_parts = explode( '.', basename( $uri ) );
@@ -102,7 +102,7 @@ class S3_Uploads_Local_Stream_Wrapper {
		$output = @chmod( $this->getLocalPath(), $mode );
		// We are modifying the underlying file here, so we have to clear the stat
		// cache so that PHP understands that URI has changed too.
		clearstatcache(TRUE, $this->getLocalPath());
		clearstatcache( true, $this->getLocalPath() );
		return $output;
	}

@@ -123,7 +123,7 @@ class S3_Uploads_Local_Stream_Wrapper {
	 *   path, as determined by the realpath() function. If $uri is set but not
	 *   valid, returns FALSE.
	 */
	protected function getLocalPath($uri = NULL) {
	protected function getLocalPath( $uri = null ) {
		if ( ! isset( $uri ) ) {
			$uri = $this->uri;
		}
@@ -133,7 +133,7 @@ class S3_Uploads_Local_Stream_Wrapper {
		$directory = realpath( $this->getDirectoryPath() );

		if ( ! $realpath || ! $directory || strpos( $realpath, $directory ) !== 0 ) {
			return FALSE;
			return false;
		}
		return $realpath;
	}
@@ -188,7 +188,7 @@ class S3_Uploads_Local_Stream_Wrapper {
			return flock( $this->handle, $operation );
		}

		return TRUE;
		return true;
	}

	/**
@@ -372,14 +372,12 @@ class S3_Uploads_Local_Stream_Wrapper {
			// $this->getLocalPath() fails if $uri has multiple levels of directories
			// that do not yet exist.
			$localpath = $this->getDirectoryPath() . '/' . $this->getTarget( $uri );
		}
		else {
		} else {
			$localpath = $this->getLocalPath( $uri );
		}
		if ( $options & STREAM_REPORT_ERRORS ) {
			return mkdir( $localpath, $mode, $recursive );
		}
		else {
		} else {
			return @mkdir( $localpath, $mode, $recursive );
		}
	}
@@ -401,8 +399,7 @@ class S3_Uploads_Local_Stream_Wrapper {
		$this->uri = $uri;
		if ( $options & STREAM_REPORT_ERRORS ) {
			return rmdir( $this->getLocalPath() );
		}
		else {
		} else {
			return @rmdir( $this->getLocalPath() );
		}
	}
@@ -428,8 +425,7 @@ class S3_Uploads_Local_Stream_Wrapper {
		// exist. This is consistent with PHP's plain filesystem stream wrapper.
		if ( $flags & STREAM_URL_STAT_QUIET || ! file_exists( $path ) ) {
			return @stat( $path );
		}
		else {
		} else {
			return stat( $path );
		}
	}
@@ -479,7 +475,7 @@ class S3_Uploads_Local_Stream_Wrapper {
		// We do not really have a way to signal a failure as rewinddir() does not
		// have a return value and there is no way to read a directory handler
		// without advancing to the next file.
		return TRUE;
		return true;
	}

	/**
@@ -494,6 +490,6 @@ class S3_Uploads_Local_Stream_Wrapper {
		closedir( $this->handle );
		// We do not really have a way to signal a failure as closedir() does not
		// have a return value.
		return TRUE;
		return true;
	}
}
+104 −58
Original line number Diff line number Diff line
<?php

use Aws\S3;

use Aws\S3\S3ClientInterface;
use Aws\CacheInterface;
use Aws\LruArrayCache;
use Aws\Result;
@@ -67,6 +68,9 @@ class S3_Uploads_Stream_Wrapper
	/** @var StreamInterface Underlying stream resource */
	private $body;

	/** @var int Size of the body that is opened */
	private $size;

	/** @var array Hash of opened stream parameters */
	private $params = [];

@@ -88,15 +92,18 @@ class S3_Uploads_Stream_Wrapper
	/** @var CacheInterface Cache for object and dir lookups */
	private $cache;

	/** @var string The opened protocol (e.g., "s3") */
	private $protocol = 's3';

	/**
	 * Register the 's3://' stream wrapper
	 *
	 * @param S3Client       $client   Client to use with the stream wrapper
	 * @param S3ClientInterface $client   Client to use with the stream wrapper
	 * @param string            $protocol Protocol to register as.
	 * @param CacheInterface    $cache    Default cache for the protocol.
	 */
	public static function register(
		Aws\S3\S3Client $client,
		S3ClientInterface $client,
		$protocol = 's3',
		CacheInterface $cache = null
	) {
@@ -126,6 +133,7 @@ class S3_Uploads_Stream_Wrapper

	public function stream_open($path, $mode, $options, &$opened_path)
	{
		$this->initProtocol($path);
		$this->params = $this->getBucketKey($path);
		$this->mode = rtrim($mode, 'bt');

@@ -145,12 +153,14 @@ class S3_Uploads_Stream_Wrapper
					 * et al that the write has failed.
					 *
					 * As a work around, we attempt to write an empty object.
					 *
					 * Added by Joe Hoyle
					 */
					try {
						$p = $this->params;
						$p['Body'] = '';
						$this->getClient()->putObject($p);
					} catch (\Exception $e) {
					} catch (Exception $e) {
						return $this->triggerError($e->getMessage());
					}

@@ -177,7 +187,7 @@ class S3_Uploads_Stream_Wrapper
		$params['Body'] = (string) $this->body;

		// Attempt to guess the ContentType of the upload based on the
		// file extension of the key
		// file extension of the key. Added by Joe Hoyle
		if (!isset($params['ContentType']) &&
			($type = Psr7\mimetype_from_filename($params['Key']))
		) {
@@ -188,7 +198,6 @@ class S3_Uploads_Stream_Wrapper
		if ( defined( 'S3_UPLOADS_HTTP_EXPIRES' ) ) {
			$params[ 'Expires' ] = S3_UPLOADS_HTTP_EXPIRES;
		}

		// Cache-Control:
		if ( defined( 'S3_UPLOADS_HTTP_CACHE_CONTROL' ) ) {
			if ( is_numeric( S3_UPLOADS_HTTP_CACHE_CONTROL ) ) {
@@ -207,9 +216,9 @@ class S3_Uploads_Stream_Wrapper
		 */
		$params = apply_filters( 's3_uploads_putObject_params',  $params );

		$this->clearCacheKey("s3://{$params['Bucket']}/{$params['Key']}");
		return $this->boolCall(function () use ($params) {
			$res = $this->getClient()->putObject($params);
			return (bool) $res;
			return (bool) $this->getClient()->putObject($params);
		});
	}

@@ -240,6 +249,8 @@ class S3_Uploads_Stream_Wrapper

	public function unlink($path)
	{
		$this->initProtocol($path);

		return $this->boolCall(function () use ($path) {
			$this->clearCacheKey($path);
			$this->getClient()->deleteObject($this->withPath($path));
@@ -250,7 +261,7 @@ class S3_Uploads_Stream_Wrapper
	public function stream_stat()
	{
		$stat = $this->getStatTemplate();
		$stat[7] = $stat['size'] = (int) $this->body->getSize();
		$stat[7] = $stat['size'] = $this->getSize();
		$stat[2] = $stat['mode'] = $this->mode;

		return $stat;
@@ -263,16 +274,18 @@ class S3_Uploads_Stream_Wrapper
	 */
	public function url_stat($path, $flags)
	{
		$extension = pathinfo($path, PATHINFO_EXTENSION);
		$this->initProtocol($path);

		$extension = pathinfo($path, PATHINFO_EXTENSION);
		/**
		 * If the file is actually just a path to a directory
		 * then return it as always existing. This is to work
		 * around wp_upload_dir doing file_exists checks on
		 * the uploads directory on every page load
		 * the uploads directory on every page load.
		 *
		 * Added by Joe Hoyle
		 */
		if ( ! $extension ) {

			return array (
					0         => 0,
					'dev'     => 0,
@@ -321,8 +334,20 @@ class S3_Uploads_Stream_Wrapper
		return $stat;
	}

	/**
	 * Parse the protocol out of the given path.
	 *
	 * @param $path
	 */
	private function initProtocol($path)
	{
		$parts = explode('://', $path, 2);
		$this->protocol = $parts[0] ?: 's3';
	}

	private function createStat($path, $flags)
	{
		$this->initProtocol($path);
		$parts = $this->withPath($path);

		if (!$parts['Key']) {
@@ -386,6 +411,7 @@ class S3_Uploads_Stream_Wrapper
	 */
	public function mkdir($path, $mode, $options)
	{
		$this->initProtocol($path);
		$params = $this->withPath($path);
		$this->clearCacheKey($path);
		if (!$params['Bucket']) {
@@ -403,6 +429,7 @@ class S3_Uploads_Stream_Wrapper

	public function rmdir($path, $options)
	{
		$this->initProtocol($path);
		$this->clearCacheKey($path);
		$params = $this->withPath($path);
		$client = $this->getClient();
@@ -437,6 +464,7 @@ class S3_Uploads_Stream_Wrapper
	 */
	public function dir_opendir($path, $options)
	{
		$this->initProtocol($path);
		$this->openedPath = $path;
		$params = $this->withPath($path);
		$delimiter = $this->getOption('delimiter');
@@ -570,6 +598,9 @@ class S3_Uploads_Stream_Wrapper
	 */
	public function rename($path_from, $path_to)
	{
		// PHP will not allow rename across wrapper types, so we can safely
		// assume $path_from and $path_to have the same protocol
		$this->initProtocol($path_from);
		$partsFrom = $this->withPath($path_from);
		$partsTo = $this->withPath($path_to);
		$this->clearCacheKey($path_from);
@@ -581,20 +612,22 @@ class S3_Uploads_Stream_Wrapper
		}

		return $this->boolCall(function () use ($partsFrom, $partsTo) {
			$options = $this->getOptions(true);
			// Copy the object and allow overriding default parameters if
			// desired, but by default copy metadata
			$this->getClient()->copyObject($this->getOptions(true) + [
				'Bucket'            => $partsTo['Bucket'],
				'Key'               => $partsTo['Key'],
				'MetadataDirective' => 'COPY',
				'CopySource'        => '/' . $partsFrom['Bucket'] . '/'
										   . rawurlencode($partsFrom['Key']),
			]);
			$this->getClient()->copy(
				$partsFrom['Bucket'],
				$partsFrom['Key'],
				$partsTo['Bucket'],
				$partsTo['Key'],
				isset($options['acl']) ? $options['acl'] : 'private',
				$options
			);
			// Delete the original object
			$this->getClient()->deleteObject([
				'Bucket' => $partsFrom['Bucket'],
				'Key'    => $partsFrom['Key']
			] + $this->getOptions(true));
			] + $options);
			return true;
		});
	}
@@ -652,11 +685,15 @@ class S3_Uploads_Stream_Wrapper
			$options = [];
		} else {
			$options = stream_context_get_options($this->context);
			$options = isset($options['s3']) ? $options['s3'] : [];
			$options = isset($options[$this->protocol])
				? $options[$this->protocol]
				: [];
		}

		$default = stream_context_get_options(stream_context_get_default());
		$default = isset($default['s3']) ? $default['s3'] : [];
		$default = isset($default[$this->protocol])
			? $default[$this->protocol]
			: [];
		$result = $this->params + $options + $default;

		if ($removeContextData) {
@@ -683,7 +720,7 @@ class S3_Uploads_Stream_Wrapper
	/**
	 * Gets the client from the stream context
	 *
	 * @return S3Client
	 * @return S3ClientInterface
	 * @throws \RuntimeException if no client has been configured
	 */
	private function getClient()
@@ -728,6 +765,7 @@ class S3_Uploads_Stream_Wrapper
		$command = $client->getCommand('GetObject', $this->getOptions(true));
		$command['@http']['stream'] = true;
		$result = $client->execute($command);
		$this->size = $result['ContentLength'];
		$this->body = $result['Body'];

		// Wrap the body in a caching entity body if seeking is allowed
@@ -980,7 +1018,15 @@ class S3_Uploads_Stream_Wrapper
		$this->getCacheStorage()->remove($key);
	}

	public function stream_metadata( $path, $option, $value ) {
		// not implemented
	/**
	 * Returns the size of the opened object body.
	 *
	 * @return int|null
	 */
	private function getSize()
	{
		$size = $this->body->getSize();

		return $size !== null ? $size : $this->size;
	}
}
Loading