Файловый менеджер - Редактировать - /var/www/html/externalstore.zip
Ðазад
PK ! tt README.mdnu �Iw�� %ExternalStore {#externalstorearch} ======================== %ExternalStore is an optional feature that enables persistent object storage outside the main database, primarily for revision text (also known as a "blob"). The main public interface for interacting with %ExternalStore is ExternalStoreAccess. Though note that higher-level concepts like {@link MediaWiki\Revision\RevisionRecord} and text blobs have their own dedicated interface: {@link MediaWiki\Revision\RevisionStore}, and {@link MediaWiki\Storage\BlobStore}. ## Concepts ### URL Objects in external stores are internally identified by a special URL. The URL is of the form `<store protocol>://<location>/<object name>`. ### Protocol The protocol represents which ExternalStoreMedium class is used. The following protocols are supported by default: - `DB`: ExternalStoreDB - `http`: ExternalStoreHttp - `mwstore`: ExternalStoreMwstore Multiple protocols may be enabled at the same time. For example, to support reading older data while using a different protocol for new data. Protocols are configured via {@link $wgExternalStores}. The ExternalStoreMedium class is decided based on concatenating the value from $wgExternalStores to the string `ExternalStore`, with a ucfirst transformation applied as-needed. A custom protocol called "foobar" could be configured by implementing ExternalStoreMedium in a subclass called `ExternalStoreFoobar`. ### Location The location identifies a particular instance of given store protocol. In the case of ExternalStoreDB, the location represents a database cluster (one or more database servers that hold the same data). When using the default of {@link Wikimedia::Rdbms::LBFactorySimple LBFactorySimple}, these clusters can be configured via {@link $wgExternalServers}. Otherwise, external clusters must be configured via {@link $wgLBFactoryConf}. ## New insertions The destination of newly stored text blobs is configured via {@link $wgDefaultExternalStore}. To enable use of %ExternalStore for new blobs, this must be set to a non-empty array. This can be disabled to store new blobs in the main database instead, it does not affect how existing blobs are read. Each destination uses a partial URL of the form `<store protocol>://<location>`. When a blob is inserted, we randomly pick an available protocol/location pair from this list. Insertions will fail-over to another default destination if the chosen one is unavailable. ## Append-only {#externalstore-appendonly} %ExternalStore is designed as an append-only system, to persist data in a way that is highly reliable and immutable. As such, the interface is restricted to fetch and insert operations, and specifically does not permit modification or deletion once data is stored. This design benefits MediaWiki in a number of ways: * The limited interface provides flexibility to each protocol implementation. * Caching is trivial and safe. * Stable references to external store can be kept outside of it, in the core database and anywhere else in caching or other storage layers, without needing to track of propagate changes. * Historical data can be stored with high reliability guarantees and operational safety: * External database clusters may be operated in read-only mode, directly through MySQL. * Each replica within the cluster may operate as independent static backup. * Database replication between hosts may be turned off. * Even command-line access from outside MediaWiki can't accidentally affect historical data. In case of maintenance tasks such as recompression, we generally iterate through known blobs and write new blobs as-needed and gracefully update pointers accordingly. If an entire cluster has been copied or recompressed to a new location, it can be taken out of rotation, with any storage space freed at that time. Note that multiple locations may be physically colocated on the same hardware, e.g. by running multiple instances of MySQL. Although it may be simpler to free space by doing recompression during other routine maintenance, such as when migrating data from old to new hardware. PK ! �H,� � ExternalStoreMemory.phpnu �Iw�� <?php /** * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. * http://www.gnu.org/copyleft/gpl.html * * @file */ /** * External storage in PHP process memory for testing. * * In this system, each store "location" is separate PHP array. * URLs are of the form "memory://location/id". The id/value pairs * at each location are segregated by DB domain ID. * * @see ExternalStoreAccess * @ingroup ExternalStorage * @since 1.33 */ class ExternalStoreMemory extends ExternalStoreMedium { /** @var array[] Map of (location => DB domain => id => value) */ private static $data = []; /** @var int */ private static $nextId = 0; public function fetchFromURL( $url ) { [ $location, $id ] = self::getURLComponents( $url ); if ( $id === null ) { throw new UnexpectedValueException( "Missing ID in URL component." ); } return self::$data[$location][$this->dbDomain][$id] ?? false; } public function batchFetchFromURLs( array $urls ) { $blobs = []; foreach ( $urls as $url ) { $blob = $this->fetchFromURL( $url ); if ( $blob !== false ) { $blobs[$url] = $blob; } } return $blobs; } public function store( $location, $data ) { $index = ++self::$nextId; self::$data[$location][$this->dbDomain][$index] = $data; return "memory://$location/$index"; } /** * Remove all data from memory for this domain */ public function clear() { foreach ( self::$data as &$dataForLocation ) { unset( $dataForLocation[$this->dbDomain] ); } unset( $dataForLocation ); self::$data = array_filter( self::$data, 'count' ); self::$nextId = 0; } /** * @param string $url * @return array (location, ID or null) */ private function getURLComponents( $url ) { // @phan-suppress-next-line PhanSuspiciousBinaryAddLists It's intentional [ $proto, $path ] = explode( '://', $url, 2 ) + [ null, null ]; if ( $proto !== 'memory' ) { throw new UnexpectedValueException( "Got URL of protocol '$proto', not 'memory'." ); } elseif ( $path === null ) { throw new UnexpectedValueException( "URL is missing path component." ); } $parts = explode( '/', $path ); if ( count( $parts ) > 2 ) { throw new UnexpectedValueException( "Too components in URL '$path'." ); } return [ $parts[0], $parts[1] ?? null ]; } } PK ! �0OK K ExternalStoreMedium.phpnu �Iw�� <?php /** * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. * http://www.gnu.org/copyleft/gpl.html * * @file */ use Psr\Log\LoggerAwareInterface; use Psr\Log\LoggerInterface; use Psr\Log\NullLogger; /** * Base class for external storage. * * There can be multiple "locations" for a storage medium type (e.g. DB clusters, filesystems). * Blobs are stored under URLs of the form `<protocol>://<location>/<path>`. Each type of storage * medium has an associated protocol. * * @see ExternalStoreAccess * @ingroup ExternalStorage * @since 1.21 */ abstract class ExternalStoreMedium implements LoggerAwareInterface { /** @var array Usage context options for this instance */ protected $params = []; /** @var string Default database domain to store content under */ protected $dbDomain; /** @var bool Whether this was factoried with an explicit DB domain */ protected $isDbDomainExplicit; /** @var LoggerInterface */ protected $logger; /** * @param array $params Usage context options for this instance: * - domain: the DB domain ID of the wiki the content is for [required] * - logger: LoggerInterface instance [optional] * - isDomainImplicit: whether this was factoried without an explicit DB domain [optional] */ public function __construct( array $params ) { $this->params = $params; if ( isset( $params['domain'] ) ) { $this->dbDomain = $params['domain']; $this->isDbDomainExplicit = empty( $params['isDomainImplicit'] ); } else { throw new InvalidArgumentException( 'Missing DB "domain" parameter.' ); } $this->logger = $params['logger'] ?? new NullLogger(); } public function setLogger( LoggerInterface $logger ) { $this->logger = $logger; } /** * Fetch data from given external store URL * * @param string $url An external store URL * @return string|bool The text stored or false on error * @throws ExternalStoreException */ abstract public function fetchFromURL( $url ); /** * Fetch data from given external store URLs. * * @param array $urls A list of external store URLs * @return string[] Map of (url => text) for the URLs where data was actually found */ public function batchFetchFromURLs( array $urls ) { $retval = []; foreach ( $urls as $url ) { $data = $this->fetchFromURL( $url ); // Dont return when false to allow for simpler implementations if ( $data !== false ) { $retval[$url] = $data; } } return $retval; } /** * Insert a data item into a given location * * @param string $location The location name * @param string $data The data item * @return string|bool The URL of the stored data item, or false on error * @throws ExternalStoreException */ abstract public function store( $location, $data ); /** * Check if a given location is read-only * * @param string $location The location name * @return bool Whether this location is read-only * @since 1.31 */ public function isReadOnly( $location ) { return false; } } PK ! �4F � � ExternalStoreFactory.phpnu �Iw�� <?php use MediaWiki\MediaWikiServices; use Psr\Log\LoggerAwareInterface; use Psr\Log\LoggerInterface; use Psr\Log\NullLogger; /** * @see ExternalStoreAccess * @internal Use the ExternalStoreAccess service instead. * @since 1.31 * @ingroup ExternalStorage */ class ExternalStoreFactory implements LoggerAwareInterface { /** @var string[] List of storage access protocols */ private $protocols; /** @var string[] List of base storage URLs that define locations for writes */ private $writeBaseUrls; /** @var string Default database domain to store content under */ private $localDomainId; /** @var LoggerInterface */ private $logger; /** @var ExternalStoreMedium[] */ private $stores = []; /** * @param string[] $externalStores See $wgExternalStores * @param string[] $defaultStores See $wgDefaultExternalStore * @param string $localDomainId Local database/wiki ID * @param LoggerInterface|null $logger */ public function __construct( array $externalStores, array $defaultStores, string $localDomainId, ?LoggerInterface $logger = null ) { $this->protocols = array_map( 'strtolower', $externalStores ); $this->writeBaseUrls = $defaultStores; $this->localDomainId = $localDomainId; $this->logger = $logger ?: new NullLogger(); } public function setLogger( LoggerInterface $logger ) { $this->logger = $logger; } /** * @return string[] List of active store types/protocols (lowercased), e.g. [ "db" ] * @since 1.34 */ public function getProtocols() { return $this->protocols; } /** * @return string[] List of default base URLs for writes, e.g. [ "DB://cluster1" ] * @since 1.34 */ public function getWriteBaseUrls() { return $this->writeBaseUrls; } /** * Get an external store object of the given type, with the given parameters * * The 'domain' field in $params will be set to the local DB domain if it is unset * or false. A special 'isDomainImplicit' flag is set when this happens, which should * only be used to handle legacy DB domain configuration concerns (e.g. T200471). * * @param string $proto Type of external storage, should be a value in $wgExternalStores * @param array $params Map of ExternalStoreMedium::__construct context parameters. * @return ExternalStoreMedium The store class or false on error * @throws ExternalStoreException When $proto is not recognized */ public function getStore( $proto, array $params = [] ) { $cacheKey = $proto . ':' . json_encode( $params ); if ( isset( $this->stores[$cacheKey] ) ) { return $this->stores[$cacheKey]; } $protoLowercase = strtolower( $proto ); // normalize if ( !$this->protocols || !in_array( $protoLowercase, $this->protocols ) ) { throw new ExternalStoreException( "Protocol '$proto' is not enabled." ); } if ( $protoLowercase === 'db' ) { $class = 'ExternalStoreDB'; } else { $class = 'ExternalStore' . ucfirst( $proto ); } if ( isset( $params['wiki'] ) ) { $params += [ 'domain' => $params['wiki'] ]; // b/c } if ( !isset( $params['domain'] ) || $params['domain'] === false ) { $params['domain'] = $this->localDomainId; // default $params['isDomainImplicit'] = true; // b/c for ExternalStoreDB } // @TODO: ideally, this class should not hardcode what classes need what backend factory // objects. For now, inject the factory instances into __construct() for those that do. if ( $protoLowercase === 'db' ) { $params['lbFactory'] = MediaWikiServices::getInstance()->getDBLoadBalancerFactory(); } elseif ( $protoLowercase === 'mwstore' ) { $params['fbGroup'] = MediaWikiServices::getInstance()->getFileBackendGroup(); } $params['logger'] = $this->logger; if ( !class_exists( $class ) ) { throw new ExternalStoreException( "Class '$class' is not defined." ); } // Any custom modules should be added to $wgAutoLoadClasses for on-demand loading $this->stores[$cacheKey] = new $class( $params ); return $this->stores[$cacheKey]; } /** * Get the ExternalStoreMedium for a given URL * * $url is either of the form: * - a) "<proto>://<location>/<path>", for retrieval, or * - b) "<proto>://<location>", for storage * * @param string $url * @param array $params Map of ExternalStoreMedium::__construct context parameters * @return ExternalStoreMedium * @throws ExternalStoreException When the protocol is missing or not recognized * @since 1.34 */ public function getStoreForUrl( $url, array $params = [] ) { [ $proto, $path ] = self::splitStorageUrl( $url ); if ( $path == '' ) { // bad URL throw new ExternalStoreException( "Invalid URL '$url'" ); } return $this->getStore( $proto, $params ); } /** * Get the location within the appropriate store for a given a URL * * @param string $url * @return string * @throws ExternalStoreException * @since 1.34 */ public function getStoreLocationFromUrl( $url ) { [ , $location ] = self::splitStorageUrl( $url ); if ( $location == '' ) { // bad URL throw new ExternalStoreException( "Invalid URL '$url'" ); } return $location; } /** * @param string[] $urls * @return string[][] Map of (protocol => list of URLs) * @throws ExternalStoreException * @since 1.34 */ public function getUrlsByProtocol( array $urls ) { $urlsByProtocol = []; foreach ( $urls as $url ) { [ $proto, ] = self::splitStorageUrl( $url ); $urlsByProtocol[$proto][] = $url; } return $urlsByProtocol; } /** * @param string $storeUrl * @return string[] (protocol, store location or location-qualified path) * @throws ExternalStoreException */ private static function splitStorageUrl( $storeUrl ) { $parts = explode( '://', $storeUrl ); if ( count( $parts ) != 2 || $parts[0] === '' || $parts[1] === '' ) { throw new ExternalStoreException( "Invalid storage URL '$storeUrl'" ); } return $parts; } } PK ! &cbb ExternalStore.phpnu �Iw�� <?php /** * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. * http://www.gnu.org/copyleft/gpl.html * * @file */ use MediaWiki\MediaWikiServices; /** * @ingroup ExternalStorage * @deprecated since 1.34 Use the ExternalStoreAccess service instead. */ class ExternalStore { /** * Get an external store object of the given type, with the given parameters * * @param string $proto Type of external storage, should be a value in $wgExternalStores * @param array $params Associative array of ExternalStoreMedium parameters * @return ExternalStoreMedium|bool The store class or false on error * @deprecated since 1.34 */ public static function getStoreObject( $proto, array $params = [] ) { try { return MediaWikiServices::getInstance() ->getExternalStoreFactory() ->getStore( $proto, $params ); } catch ( ExternalStoreException $e ) { return false; } } /** * Fetch data from given URL * * @param string $url The URL of the text to get * @param array $params Associative array of ExternalStoreMedium parameters * @return string|bool The text stored or false on error * @deprecated since 1.34 */ public static function fetchFromURL( $url, array $params = [] ) { try { return MediaWikiServices::getInstance() ->getExternalStoreAccess() ->fetchFromURL( $url, $params ); } catch ( ExternalStoreException $e ) { return false; } } /** * Store a data item to an external store, identified by a partial URL * The protocol part is used to identify the class, the rest is passed to the * class itself as a parameter. * * @param string $url A partial external store URL ("<store type>://<location>") * @param string $data * @param array $params Associative array of ExternalStoreMedium parameters * @return string|bool The URL of the stored data item, or false on error * @deprecated since 1.34 */ public static function insert( $url, $data, array $params = [] ) { try { $esFactory = MediaWikiServices::getInstance()->getExternalStoreFactory(); $location = $esFactory->getStoreLocationFromUrl( $url ); return $esFactory->getStoreForUrl( $url, $params )->store( $location, $data ); } catch ( ExternalStoreException $e ) { return false; } } /** * Fetch data from multiple URLs with a minimum of round trips * * @param array $urls The URLs of the text to get * @return array Map from url to its data. Data is either string when found * or false on failure. * @throws ExternalStoreException * @deprecated since 1.34 */ public static function batchFetchFromURLs( array $urls ) { return MediaWikiServices::getInstance()->getExternalStoreAccess()->fetchFromURLs( $urls ); } /** * Like insert() above, but does more of the work for us. * This function does not need a url param, it builds it by * itself. It also fails-over to the next possible clusters * provided by $wgDefaultExternalStore. * * @param string $data * @param array $params Map of ExternalStoreMedium::__construct context parameters * @return string The URL of the stored data item * @throws ExternalStoreException * @deprecated since 1.34 */ public static function insertToDefault( $data, array $params = [] ) { return MediaWikiServices::getInstance()->getExternalStoreAccess()->insert( $data, $params ); } /** * Like insert() above, but does more of the work for us. * This function does not need a url param, it builds it by * itself. It also fails-over to the next possible clusters * as provided in the first parameter. * * @param array $tryStores Refer to $wgDefaultExternalStore * @param string $data * @param array $params Map of ExternalStoreMedium::__construct context parameters * @return string The URL of the stored data item * @throws ExternalStoreException * @deprecated since 1.34 */ public static function insertWithFallback( array $tryStores, $data, array $params = [] ) { return MediaWikiServices::getInstance() ->getExternalStoreAccess() ->insert( $data, $params, $tryStores ); } /** * @param string $data * @param string $wiki * @return string The URL of the stored data item * @throws ExternalStoreException * @deprecated since 1.34 Use insertToDefault() with 'wiki' set */ public static function insertToForeignDefault( $data, $wiki ) { return MediaWikiServices::getInstance() ->getExternalStoreAccess() ->insert( $data, [ 'domain' => $wiki ] ); } } PK ! �� �@ @ ExternalStoreMwstore.phpnu �Iw�� <?php /** * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. * http://www.gnu.org/copyleft/gpl.html * * @file */ use MediaWiki\FileBackend\FileBackendGroup; use MediaWiki\MediaWikiServices; use MediaWiki\WikiMap\WikiMap; use Wikimedia\FileBackend\FileBackend; use Wikimedia\FileBackend\FSFileBackend; /** * External storage in a FileBackend. * * In this system, each store "location" maps to the name of a file backend. * The file backends must be defined in $wgFileBackends and must be global * and fully qualified with a global "wikiId" prefix in the configuration. * * @see ExternalStoreAccess * @ingroup ExternalStorage * @since 1.21 */ class ExternalStoreMwstore extends ExternalStoreMedium { /** @var FileBackendGroup */ private $fbGroup; /** * @see ExternalStoreMedium::__construct() * @param array $params Additional parameters include: * - fbGroup: a FileBackendGroup instance */ public function __construct( array $params ) { parent::__construct( $params ); if ( !isset( $params['fbGroup'] ) || !( $params['fbGroup'] instanceof FileBackendGroup ) ) { throw new InvalidArgumentException( "FileBackendGroup required in 'fbGroup' field." ); } $this->fbGroup = $params['fbGroup']; } /** * Fetch data from a given external store URL * * @see ExternalStoreMedium::fetchFromURL() * @param string $url An external store URL in the form of mwstore://backend/container/wiki/id * @return string|bool */ public function fetchFromURL( $url ) { $be = $this->fbGroup->backendFromPath( $url ); if ( $be instanceof FileBackend ) { // We don't need "latest" since objects are immutable and // backends should at least have "read-after-create" consistency. return $be->getFileContents( [ 'src' => $url ] ); } return false; } /** * Fetch data from given external store URLs. * The URLs are in the form of mwstore://backend/container/wiki/id * * @param array $urls An array of external store URLs * @return array A map from url to stored content. Failed results are not represented. */ public function batchFetchFromURLs( array $urls ) { $pathsByBackend = []; foreach ( $urls as $url ) { $be = $this->fbGroup->backendFromPath( $url ); if ( $be instanceof FileBackend ) { $pathsByBackend[$be->getName()][] = $url; } } $blobs = []; foreach ( $pathsByBackend as $backendName => $paths ) { $be = $this->fbGroup->get( $backendName ); $blobs += $be->getFileContentsMulti( [ 'srcs' => $paths ] ); } return $blobs; } public function store( $backend, $data ) { $be = $this->fbGroup->get( $backend ); // Get three random base 36 characters to act as shard directories $rand = Wikimedia\base_convert( (string)mt_rand( 0, 46655 ), 10, 36, 3 ); // Make sure ID is roughly lexicographically increasing for performance $gen = MediaWikiServices::getInstance()->getGlobalIdGenerator(); $id = str_pad( $gen->newTimestampedUID128( 32 ), 26, '0', STR_PAD_LEFT ); // Segregate items by DB domain ID for the sake of bookkeeping $domain = $this->isDbDomainExplicit ? $this->dbDomain // @FIXME: this does not include the schema for b/c but it ideally should : WikiMap::getWikiIdFromDbDomain( $this->dbDomain ); $url = $be->getContainerStoragePath( 'data' ) . '/' . rawurlencode( $domain ); // Use directory/container sharding $url .= ( $be instanceof FSFileBackend ) ? "/{$rand[0]}/{$rand[1]}/{$rand[2]}/{$id}" // keep directories small : "/{$rand[0]}/{$rand[1]}/{$id}"; // container sharding is only 2-levels $be->prepare( [ 'dir' => dirname( $url ), 'noAccess' => 1, 'noListing' => 1 ] ); $status = $be->create( [ 'dst' => $url, 'content' => $data ] ); if ( $status->isOK() ) { return $url; } throw new ExternalStoreException( __METHOD__ . ": operation failed: $status" ); } public function isReadOnly( $backend ) { if ( parent::isReadOnly( $backend ) ) { return true; } $be = $this->fbGroup->get( $backend ); return $be->isReadOnly(); } } PK ! T� �� � ExternalStoreHttp.phpnu �Iw�� <?php /** * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. * http://www.gnu.org/copyleft/gpl.html * * @file */ use MediaWiki\MediaWikiServices; /** * External storage using HTTP requests. * * Example class for HTTP accessible external objects. * Only supports reading, not storing. * * @see ExternalStoreAccess * @ingroup ExternalStorage */ class ExternalStoreHttp extends ExternalStoreMedium { public function fetchFromURL( $url ) { return MediaWikiServices::getInstance()->getHttpRequestFactory()-> get( $url, [], __METHOD__ ); } public function store( $location, $data ) { // @phan-suppress-previous-line PhanPluginNeverReturnMethod throw new LogicException( "ExternalStoreHttp is read-only and does not support store()." ); } public function isReadOnly( $location ) { return true; } } PK ! � �$�1 �1 ExternalStoreDB.phpnu �Iw�� <?php /** * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. * http://www.gnu.org/copyleft/gpl.html * * @file */ use Wikimedia\Rdbms\DatabaseDomain; use Wikimedia\Rdbms\DBUnexpectedError; use Wikimedia\Rdbms\ILoadBalancer; use Wikimedia\Rdbms\LBFactory; use Wikimedia\Rdbms\Query; use Wikimedia\Rdbms\ServerInfo; use Wikimedia\ScopedCallback; /** * External storage in a SQL database. * * In this system, each store "location" maps to a database "cluster". * The clusters must be defined in the normal LBFactory configuration. * * @see ExternalStoreAccess * @ingroup ExternalStorage */ class ExternalStoreDB extends ExternalStoreMedium { /** @var LBFactory */ private $lbFactory; /** * @see ExternalStoreMedium::__construct() * @param array $params Additional parameters include: * - lbFactory: an LBFactory instance */ public function __construct( array $params ) { parent::__construct( $params ); if ( !isset( $params['lbFactory'] ) || !( $params['lbFactory'] instanceof LBFactory ) ) { throw new InvalidArgumentException( "LBFactory required in 'lbFactory' field." ); } $this->lbFactory = $params['lbFactory']; } /** * Fetch data from given external store URL. * * The provided URL is in the form of `DB://cluster/id` or `DB://cluster/id/itemid` * for concatenated storage if ConcatenatedGzipHistoryBlob was used. * * @param string $url * @return string|false False if missing * @see ExternalStoreMedium::fetchFromURL() */ public function fetchFromURL( $url ) { [ $cluster, $id, $itemID ] = $this->parseURL( $url ); $ret = $this->fetchBlob( $cluster, $id, $itemID ); if ( $itemID !== false && $ret !== false ) { return $ret->getItem( $itemID ); } return $ret; } /** * Fetch multiple URLs from given external store. * * The provided URLs are in the form of `DB://cluster/id`, or `DB://cluster/id/itemid` * for concatenated storage if ConcatenatedGzipHistoryBlob was used. * * @param array $urls An array of external store URLs * @return array A map from url to stored content. Failed results * are not represented. */ public function batchFetchFromURLs( array $urls ) { $batched = $inverseUrlMap = []; foreach ( $urls as $url ) { [ $cluster, $id, $itemID ] = $this->parseURL( $url ); $batched[$cluster][$id][] = $itemID; // false $itemID gets cast to int, but should be ok // since we do === from the $itemID in $batched $inverseUrlMap[$cluster][$id][$itemID] = $url; } $ret = []; foreach ( $batched as $cluster => $batchByCluster ) { $res = $this->batchFetchBlobs( $cluster, $batchByCluster ); /** @var HistoryBlob $blob */ foreach ( $res as $id => $blob ) { foreach ( $batchByCluster[$id] as $itemID ) { $url = $inverseUrlMap[$cluster][$id][$itemID]; if ( $itemID === false ) { $ret[$url] = $blob; } else { $ret[$url] = $blob->getItem( $itemID ); } } } } return $ret; } /** * @inheritDoc */ public function store( $location, $data ) { $blobsTable = $this->getTable( $location ); $dbw = $this->getPrimary( $location ); $dbw->newInsertQueryBuilder() ->insertInto( $blobsTable ) ->row( [ 'blob_text' => $data ] ) ->caller( __METHOD__ )->execute(); $id = $dbw->insertId(); if ( !$id ) { throw new ExternalStoreException( __METHOD__ . ': no insert ID' ); } return "DB://$location/$id"; } /** * @inheritDoc */ public function isReadOnly( $location ) { if ( parent::isReadOnly( $location ) ) { return true; } return ( $this->getLoadBalancer( $location )->getReadOnlyReason() !== false ); } /** * Get a LoadBalancer for the specified cluster * * @param string $cluster Cluster name * @return ILoadBalancer */ private function getLoadBalancer( $cluster ) { return $this->lbFactory->getExternalLB( $cluster ); } /** * Get a replica DB connection for the specified cluster * * @since 1.34 * @param string $cluster Cluster name * @return \Wikimedia\Rdbms\IReadableDatabase */ public function getReplica( $cluster ) { $lb = $this->getLoadBalancer( $cluster ); return $lb->getConnection( DB_REPLICA, [], $this->getDomainId( $lb->getServerInfo( ServerInfo::WRITER_INDEX ) ), $lb::CONN_TRX_AUTOCOMMIT ); } /** * Get a primary database connection for the specified cluster * * @param string $cluster Cluster name * @return \Wikimedia\Rdbms\IMaintainableDatabase * @since 1.37 */ public function getPrimary( $cluster ) { $lb = $this->getLoadBalancer( $cluster ); return $lb->getMaintenanceConnectionRef( DB_PRIMARY, [], $this->getDomainId( $lb->getServerInfo( ServerInfo::WRITER_INDEX ) ), $lb::CONN_TRX_AUTOCOMMIT ); } /** * @param array $server Primary DB server configuration array for LoadBalancer * @return string|false Database domain ID or false */ private function getDomainId( array $server ) { if ( $this->isDbDomainExplicit ) { return $this->dbDomain; // explicit foreign domain } if ( isset( $server['dbname'] ) ) { // T200471: for b/c, treat any "dbname" field as forcing which database to use. // MediaWiki/LoadBalancer previously did not enforce any concept of a local DB // domain, but rather assumed that the LB server configuration matched $wgDBname. // This check is useful when the external storage DB for this cluster does not use // the same name as the corresponding "main" DB(s) for wikis. $domain = new DatabaseDomain( $server['dbname'], $server['schema'] ?? null, $server['tablePrefix'] ?? '' ); return $domain->getId(); } return false; // local LB domain } /** * Get the configured blobs table name for this database * * Typically, a suffix like "_clusterX" can be used to facilitate clean merging of * read-only storage clusters by simply cloning tables to the new cluster servers. * * @param string $cluster Cluster name * @return string Unqualified table name (e.g. "blobs_cluster32" or default "blobs") * @internal Only for use within ExternalStoreDB and its core maintenance scripts */ public function getTable( string $cluster ) { $lb = $this->getLoadBalancer( $cluster ); $info = $lb->getServerInfo( ServerInfo::WRITER_INDEX ); return $info['blobs table'] ?? 'blobs'; } /** * Create the appropriate blobs table on this cluster * * @since 1.34 * @param string $cluster */ public function initializeTable( $cluster ) { global $IP; static $supportedTypes = [ 'mysql', 'sqlite' ]; $dbw = $this->getPrimary( $cluster ); if ( !in_array( $dbw->getType(), $supportedTypes, true ) ) { throw new DBUnexpectedError( $dbw, "RDBMS type '{$dbw->getType()}' not supported." ); } $sqlFilePath = "$IP/maintenance/storage/blobs.sql"; $sql = file_get_contents( $sqlFilePath ); if ( $sql === false ) { throw new RuntimeException( "Failed to read '$sqlFilePath'." ); } $blobsTable = $this->getTable( $cluster ); $encTable = $dbw->tableName( $blobsTable ); $sqlWithReplacedVars = str_replace( [ '/*$wgDBprefix*/blobs', '/*_*/blobs' ], [ $encTable, $encTable ], $sql ); $dbw->query( new Query( $sqlWithReplacedVars, $dbw::QUERY_CHANGE_SCHEMA, 'CREATE', $blobsTable, $sqlWithReplacedVars ), __METHOD__ ); } /** * Fetch a blob item out of the database; a cache of the last-loaded * blob will be kept so that multiple loads out of a multi-item blob * can avoid redundant database access and decompression. * @param string $cluster * @param string $id * @param string $itemID * @return HistoryBlob|false Returns false if missing */ private function fetchBlob( $cluster, $id, $itemID ) { /** * One-step cache variable to hold base blobs; operations that * pull multiple revisions may often pull multiple times from * the same blob. By keeping the last-used one open, we avoid * redundant unserialization and decompression overhead. */ static $externalBlobCache = []; $cacheID = ( $itemID === false ) ? "$cluster/$id" : "$cluster/$id/"; $cacheID = "$cacheID@{$this->dbDomain}"; if ( isset( $externalBlobCache[$cacheID] ) ) { $this->logger->debug( __METHOD__ . ": cache hit on $cacheID" ); return $externalBlobCache[$cacheID]; } $this->logger->debug( __METHOD__ . ": cache miss on $cacheID" ); $blobsTable = $this->getTable( $cluster ); $dbr = $this->getReplica( $cluster ); $ret = $dbr->newSelectQueryBuilder() ->select( 'blob_text' ) ->from( $blobsTable ) ->where( [ 'blob_id' => $id ] ) ->caller( __METHOD__ )->fetchField(); if ( $ret === false ) { // Try the primary DB $this->logger->warning( __METHOD__ . ": primary DB fallback on $cacheID" ); $trxProfiler = $this->lbFactory->getTransactionProfiler(); $scope = $trxProfiler->silenceForScope( $trxProfiler::EXPECTATION_REPLICAS_ONLY ); $dbw = $this->getPrimary( $cluster ); $ret = $dbw->newSelectQueryBuilder() ->select( 'blob_text' ) ->from( $blobsTable ) ->where( [ 'blob_id' => $id ] ) ->caller( __METHOD__ )->fetchField(); ScopedCallback::consume( $scope ); if ( $ret === false ) { $this->logger->warning( __METHOD__ . ": primary DB failed to find $cacheID" ); } } if ( $itemID !== false && $ret !== false ) { // Unserialise object; caller extracts item $ret = HistoryBlobUtils::unserialize( $ret ); } $externalBlobCache = [ $cacheID => $ret ]; return $ret; } /** * Fetch multiple blob items out of the database * * @param string $cluster A cluster name valid for use with LBFactory * @param array $ids A map from the blob_id's to look for to the requested itemIDs in the blobs * @return array A map from the blob_id's requested to their content. * Unlocated ids are not represented */ private function batchFetchBlobs( $cluster, array $ids ) { $blobsTable = $this->getTable( $cluster ); $dbr = $this->getReplica( $cluster ); $res = $dbr->newSelectQueryBuilder() ->select( [ 'blob_id', 'blob_text' ] ) ->from( $blobsTable ) ->where( [ 'blob_id' => array_keys( $ids ) ] ) ->caller( __METHOD__ ) ->fetchResultSet(); $ret = []; $this->mergeBatchResult( $ret, $ids, $res ); if ( $ids ) { // Try the primary $this->logger->info( __METHOD__ . ": primary fallback on '$cluster' for: " . implode( ',', array_keys( $ids ) ) ); $trxProfiler = $this->lbFactory->getTransactionProfiler(); $scope = $trxProfiler->silenceForScope( $trxProfiler::EXPECTATION_REPLICAS_ONLY ); $dbw = $this->getPrimary( $cluster ); $res = $dbw->newSelectQueryBuilder() ->select( [ 'blob_id', 'blob_text' ] ) ->from( $blobsTable ) ->where( [ 'blob_id' => array_keys( $ids ) ] ) ->caller( __METHOD__ ) ->fetchResultSet(); ScopedCallback::consume( $scope ); $this->mergeBatchResult( $ret, $ids, $res ); } if ( $ids ) { $this->logger->error( __METHOD__ . ": primary on '$cluster' failed locating items: " . implode( ',', array_keys( $ids ) ) ); } return $ret; } /** * Helper function for self::batchFetchBlobs for merging primary/replica DB results * @param array &$ret Current self::batchFetchBlobs return value * @param array &$ids Map from blob_id to requested itemIDs * @param mixed $res DB result from Database::select */ private function mergeBatchResult( array &$ret, array &$ids, $res ) { foreach ( $res as $row ) { $id = $row->blob_id; $itemIDs = $ids[$id]; unset( $ids[$id] ); // to track if everything is found if ( count( $itemIDs ) === 1 && reset( $itemIDs ) === false ) { // single result stored per blob $ret[$id] = $row->blob_text; } else { // multi result stored per blob $ret[$id] = HistoryBlobUtils::unserialize( $row->blob_text ); } } } /** * @param string $url * @return array */ protected function parseURL( $url ) { $path = explode( '/', $url ); return [ $path[2], // cluster $path[3], // id $path[4] ?? false // itemID ]; } } PK ! �x�bk k ExternalStoreException.phpnu �Iw�� <?php /** * @newable * @ingroup ExternalStorage */ class ExternalStoreException extends Exception { } PK ! �n0n� � ExternalStoreAccess.phpnu �Iw�� <?php /** * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. * http://www.gnu.org/copyleft/gpl.html * * @file */ use Psr\Log\LoggerAwareInterface; use Psr\Log\LoggerInterface; use Psr\Log\NullLogger; use Wikimedia\RequestTimeout\TimeoutException; /** * @defgroup ExternalStorage ExternalStorage * * Object storage outside the main database, see also [ExternalStore Architecture](@ref externalstorearch). */ /** * This is the main interface for fetching or inserting objects with [ExternalStore](@ref externalstorearch). * * This interface is meant to mimic the ExternalStoreMedium base class (which * represents a single external store protocol), and transparently uses the * right instance of that class when fetching by URL. * * @see [ExternalStore Architecture](@ref externalstorearch). * @ingroup ExternalStorage * @since 1.34 */ class ExternalStoreAccess implements LoggerAwareInterface { /** @var ExternalStoreFactory */ private $storeFactory; /** @var LoggerInterface */ private $logger; /** * @param ExternalStoreFactory $factory * @param LoggerInterface|null $logger */ public function __construct( ExternalStoreFactory $factory, ?LoggerInterface $logger = null ) { $this->storeFactory = $factory; $this->logger = $logger ?: new NullLogger(); } public function setLogger( LoggerInterface $logger ) { $this->logger = $logger; } /** * Fetch data from given URL * * @see ExternalStoreFactory::getStore() * * @param string $url The URL of the text to get * @param array $params Map of context parameters; same as ExternalStoreFactory::getStore() * @return string|false The text stored or false on error * @throws ExternalStoreException */ public function fetchFromURL( $url, array $params = [] ) { return $this->storeFactory->getStoreForUrl( $url, $params )->fetchFromURL( $url ); } /** * Fetch data from multiple URLs with a minimum of round trips * * @see ExternalStoreFactory::getStore() * * @param array $urls The URLs of the text to get * @param array $params Map of context parameters; same as ExternalStoreFactory::getStore() * @return array Map of (url => string or false if not found) * @throws ExternalStoreException */ public function fetchFromURLs( array $urls, array $params = [] ) { $batches = $this->storeFactory->getUrlsByProtocol( $urls ); $retval = []; foreach ( $batches as $proto => $batchedUrls ) { $store = $this->storeFactory->getStore( $proto, $params ); $retval += $store->batchFetchFromURLs( $batchedUrls ); } // invalid, not found, db dead, etc. $missing = array_diff( $urls, array_keys( $retval ) ); foreach ( $missing as $url ) { $retval[$url] = false; } return $retval; } /** * Insert data into storage and return the assigned URL * * This will randomly pick one of the available write storage locations to put the data. * It will keep failing-over to any untried storage locations whenever one location is * not usable. * * @see ExternalStoreFactory::getStore() * * @param string $data * @param array $params Map of context parameters; same as ExternalStoreFactory::getStore() * @param string[]|null $tryStores Base URLs to try, e.g. [ "DB://cluster1" ] * @return string|false The URL of the stored data item, or false on error * @throws ExternalStoreException */ public function insert( $data, array $params = [], ?array $tryStores = null ) { $tryStores ??= $this->storeFactory->getWriteBaseUrls(); if ( !$tryStores ) { throw new ExternalStoreException( "List of external stores provided is empty." ); } $error = false; // track the last exception thrown $readOnlyCount = 0; // track if a store was read-only while ( count( $tryStores ) > 0 ) { $index = mt_rand( 0, count( $tryStores ) - 1 ); $storeUrl = $tryStores[$index]; $this->logger->debug( __METHOD__ . ": trying $storeUrl" ); $store = $this->storeFactory->getStoreForUrl( $storeUrl, $params ); if ( $store === false ) { throw new ExternalStoreException( "Invalid external storage protocol - $storeUrl" ); } $location = $this->storeFactory->getStoreLocationFromUrl( $storeUrl ); try { if ( $store->isReadOnly( $location ) ) { $readOnlyCount++; $msg = 'read only'; } else { $url = $store->store( $location, $data ); if ( strlen( $url ) ) { // A store accepted the write; done! return $url; } throw new ExternalStoreException( "No URL returned by storage medium ($storeUrl)" ); } } catch ( TimeoutException $e ) { throw $e; } catch ( Exception $ex ) { $error = $ex; $msg = 'caught ' . get_class( $error ) . ' exception: ' . $error->getMessage(); } unset( $tryStores[$index] ); // Don't try this one again! $tryStores = array_values( $tryStores ); // Must have consecutive keys $this->logger->error( "Unable to store text to external storage {store_path} ({failure})", [ 'store_path' => $storeUrl, 'failure' => $msg ] ); } // We only get here when all stores failed. if ( $error ) { // At least one store threw an exception. Re-throw the most recent one. throw $error; } elseif ( $readOnlyCount ) { // If no exceptions where thrown and we get here, // this should mean that all stores were in read-only mode. throw new ReadOnlyError(); } else { // We shouldn't get here. If there were no failures, this method should have returned // from inside the body of the loop. throw new LogicException( "Unexpected failure to store text to external store" ); } } /** * @param string[]|string|null $storeUrls Base URL(s) to check, e.g. [ "DB://cluster1" ] * @return bool Whether all the default insertion stores are marked as read-only * @throws ExternalStoreException */ public function isReadOnly( $storeUrls = null ) { if ( $storeUrls === null ) { $storeUrls = $this->storeFactory->getWriteBaseUrls(); } else { $storeUrls = is_array( $storeUrls ) ? $storeUrls : [ $storeUrls ]; } if ( !$storeUrls ) { return false; // no stores exists which can be "read only" } foreach ( $storeUrls as $storeUrl ) { $store = $this->storeFactory->getStoreForUrl( $storeUrl ); $location = $this->storeFactory->getStoreLocationFromUrl( $storeUrl ); if ( $store !== false && !$store->isReadOnly( $location ) ) { return false; // at least one store is not read-only } } return true; // all stores are read-only } } PK ! tt README.mdnu �Iw�� PK ! �H,� � M ExternalStoreMemory.phpnu �Iw�� PK ! �0OK K ExternalStoreMedium.phpnu �Iw�� PK ! �4F � � �* ExternalStoreFactory.phpnu �Iw�� PK ! &cbb �A ExternalStore.phpnu �Iw�� PK ! �� �@ @ V ExternalStoreMwstore.phpnu �Iw�� PK ! T� �� � �h ExternalStoreHttp.phpnu �Iw�� PK ! � �$�1 �1 �n ExternalStoreDB.phpnu �Iw�� PK ! �x�bk k �� ExternalStoreException.phpnu �Iw�� PK ! �n0n� � J� ExternalStoreAccess.phpnu �Iw�� PK Q |�
| ver. 1.1 | |
.
| PHP 8.4.18 | Ð“ÐµÐ½ÐµÑ€Ð°Ñ†Ð¸Ñ Ñтраницы: 0 |
proxy
|
phpinfo
|
ÐаÑтройка