Changing front

This commit is contained in:
ewandor committed 2023-01-16 17:49:38 +01:00
1 parent 0b8a93b256
commit 4fe4be7730
48586 files changed
+4725790 -17464

No files matched your search

+16
View File
@@ -0,0 +1,16 @@
ISC License
Copyright 2017-2022 (c) npm, Inc.
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE COPYRIGHT HOLDER DISCLAIMS
ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE
USE OR PERFORMANCE OF THIS SOFTWARE.
+361
View File
@@ -0,0 +1,361 @@
# make-fetch-happen
[![npm version](https://img.shields.io/npm/v/make-fetch-happen.svg)](https://npm.im/make-fetch-happen) [![license](https://img.shields.io/npm/l/make-fetch-happen.svg)](https://npm.im/make-fetch-happen) [![Travis](https://img.shields.io/travis/npm/make-fetch-happen.svg)](https://travis-ci.org/npm/make-fetch-happen) [![Coverage Status](https://coveralls.io/repos/github/npm/make-fetch-happen/badge.svg?branch=latest)](https://coveralls.io/github/npm/make-fetch-happen?branch=latest)
[`make-fetch-happen`](https://github.com/npm/make-fetch-happen) is a Node.js
library that wraps [`minipass-fetch`](https://github.com/npm/minipass-fetch) with additional
features [`minipass-fetch`](https://github.com/npm/minipass-fetch) doesn't intend to include, including HTTP Cache support, request
pooling, proxies, retries, [and more](#features)!
## Install
`$ npm install --save make-fetch-happen`
## Table of Contents
* [Example](#example)
* [Features](#features)
* [Contributing](#contributing)
* [API](#api)
* [`fetch`](#fetch)
* [`fetch.defaults`](#fetch-defaults)
* [`minipass-fetch` options](#minipass-fetch-options)
* [`make-fetch-happen` options](#extra-options)
* [`opts.cachePath`](#opts-cache-path)
* [`opts.cache`](#opts-cache)
* [`opts.proxy`](#opts-proxy)
* [`opts.noProxy`](#opts-no-proxy)
* [`opts.ca, opts.cert, opts.key`](#https-opts)
* [`opts.maxSockets`](#opts-max-sockets)
* [`opts.retry`](#opts-retry)
* [`opts.onRetry`](#opts-onretry)
* [`opts.integrity`](#opts-integrity)
* [`opts.dns`](#opts-dns)
* [Message From Our Sponsors](#wow)
### Example
```javascript
const fetch = require('make-fetch-happen').defaults({
cachePath: './my-cache' // path where cache will be written (and read)
})
fetch('https://registry.npmjs.org/make-fetch-happen').then(res => {
return res.json() // download the body as JSON
}).then(body => {
console.log(`got ${body.name} from web`)
return fetch('https://registry.npmjs.org/make-fetch-happen', {
cache: 'no-cache' // forces a conditional request
})
}).then(res => {
console.log(res.status) // 304! cache validated!
return res.json().then(body => {
console.log(`got ${body.name} from cache`)
})
})
```
### Features
* Builds around [`minipass-fetch`](https://npm.im/minipass-fetch) for the core [`fetch` API](https://fetch.spec.whatwg.org) implementation
* Request pooling out of the box
* Quite fast, really
* Automatic HTTP-semantics-aware request retries
* Cache-fallback automatic "offline mode"
* Proxy support (http, https, socks, socks4, socks5)
* Built-in request caching following full HTTP caching rules (`Cache-Control`, `ETag`, `304`s, cache fallback on error, etc).
* Node.js Stream support
* Transparent gzip and deflate support
* [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) support
* Literally punches nazis
* Built in DNS cache
* (PENDING) Range request caching and resuming
### Contributing
The make-fetch-happen team enthusiastically welcomes contributions and project participation! There's a bunch of things you can do if you want to contribute! The [Contributor Guide](https://github.com/npm/cli/blob/latest/CONTRIBUTING.md) outlines the process for community interaction and contribution. Please don't hesitate to jump in if you'd like to, or even ask us questions if something isn't clear.
All participants and maintainers in this project are expected to follow the [npm Code of Conduct](https://www.npmjs.com/policies/conduct), and just generally be excellent to each other.
Please refer to the [Changelog](CHANGELOG.md) for project history details, too.
Happy hacking!
### API
#### <a name="fetch"></a> `> fetch(uriOrRequest, [opts]) -> Promise<Response>`
This function implements most of the [`fetch` API](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch): given a `uri` string or a `Request` instance, it will fire off an http request and return a Promise containing the relevant response.
If `opts` is provided, the [`minipass-fetch`-specific options](#minipass-fetch-options) will be passed to that library. There are also [additional options](#extra-options) specific to make-fetch-happen that add various features, such as HTTP caching, integrity verification, proxy support, and more.
##### Example
```javascript
fetch('https://google.com').then(res => res.buffer())
```
#### <a name="fetch-defaults"></a> `> fetch.defaults([defaultUrl], [defaultOpts])`
Returns a new `fetch` function that will call `make-fetch-happen` using `defaultUrl` and `defaultOpts` as default values to any calls.
A defaulted `fetch` will also have a `.defaults()` method, so they can be chained.
##### Example
```javascript
const fetch = require('make-fetch-happen').defaults({
cachePath: './my-local-cache'
})
fetch('https://registry.npmjs.org/make-fetch-happen') // will always use the cache
```
#### <a name="minipass-fetch-options"></a> `> minipass-fetch options`
The following options for `minipass-fetch` are used as-is:
* method
* body
* redirect
* follow
* timeout
* compress
* size
These other options are modified or augmented by make-fetch-happen:
* headers - Default `User-Agent` set to make-fetch happen. `Connection` is set to `keep-alive` or `close` automatically depending on `opts.agent`.
* agent
* If agent is null, an http or https Agent will be automatically used. By default, these will be `http.globalAgent` and `https.globalAgent`.
* If [`opts.proxy`](#opts-proxy) is provided and `opts.agent` is null, the agent will be set to an appropriate proxy-handling agent.
* If `opts.agent` is an object, it will be used as the request-pooling agent argument for this request.
* If `opts.agent` is `false`, it will be passed as-is to the underlying request library. This causes a new Agent to be spawned for every request.
For more details, see [the documentation for `minipass-fetch` itself](https://github.com/npm/minipass-fetch#options).
#### <a name="extra-options"></a> `> make-fetch-happen options`
make-fetch-happen augments the `minipass-fetch` API with additional features available through extra options. The following extra options are available:
* [`opts.cachePath`](#opts-cache-path) - Cache target to read/write
* [`opts.cache`](#opts-cache) - `fetch` cache mode. Controls cache *behavior*.
* [`opts.proxy`](#opts-proxy) - Proxy agent
* [`opts.noProxy`](#opts-no-proxy) - Domain segments to disable proxying for.
* [`opts.ca, opts.cert, opts.key, opts.strictSSL`](#https-opts)
* [`opts.localAddress`](#opts-local-address)
* [`opts.maxSockets`](#opts-max-sockets)
* [`opts.retry`](#opts-retry) - Request retry settings
* [`opts.onRetry`](#opts-onretry) - a function called whenever a retry is attempted
* [`opts.integrity`](#opts-integrity) - [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) metadata.
* [`opts.dns`](#opts-dns) - DNS cache options
#### <a name="opts-cache-path"></a> `> opts.cachePath`
A string `Path` to be used as the cache root for [`cacache`](https://npm.im/cacache).
**NOTE**: Requests will not be cached unless their response bodies are consumed. You will need to use one of the `res.json()`, `res.buffer()`, etc methods on the response, or drain the `res.body` stream, in order for it to be written.
The default cache manager also adds the following headers to cached responses:
* `X-Local-Cache`: Path to the cache the content was found in
* `X-Local-Cache-Key`: Unique cache entry key for this response
* `X-Local-Cache-Mode`: Always `stream` to indicate how the response was read from cacache
* `X-Local-Cache-Hash`: Specific integrity hash for the cached entry
* `X-Local-Cache-Status`: One of `miss`, `hit`, `stale`, `revalidated`, `updated`, or `skip` to signal how the response was created
* `X-Local-Cache-Time`: UTCString of the cache insertion time for the entry
Using [`cacache`](https://npm.im/cacache), a call like this may be used to
manually fetch the cached entry:
```javascript
const h = response.headers
cacache.get(h.get('x-local-cache'), h.get('x-local-cache-key'))
// grab content only, directly:
cacache.get.byDigest(h.get('x-local-cache'), h.get('x-local-cache-hash'))
```
##### Example
```javascript
fetch('https://registry.npmjs.org/make-fetch-happen', {
cachePath: './my-local-cache'
}) // -> 200-level response will be written to disk
```
#### <a name="opts-cache"></a> `> opts.cache`
This option follows the standard `fetch` API cache option. This option will do nothing if [`opts.cachePath`](#opts-cache-path) is null. The following values are accepted (as strings):
* `default` - Fetch will inspect the HTTP cache on the way to the network. If there is a fresh response it will be used. If there is a stale response a conditional request will be created, and a normal request otherwise. It then updates the HTTP cache with the response. If the revalidation request fails (for example, on a 500 or if you're offline), the stale response will be returned.
* `no-store` - Fetch behaves as if there is no HTTP cache at all.
* `reload` - Fetch behaves as if there is no HTTP cache on the way to the network. Ergo, it creates a normal request and updates the HTTP cache with the response.
* `no-cache` - Fetch creates a conditional request if there is a response in the HTTP cache and a normal request otherwise. It then updates the HTTP cache with the response.
* `force-cache` - Fetch uses any response in the HTTP cache matching the request, not paying attention to staleness. If there was no response, it creates a normal request and updates the HTTP cache with the response.
* `only-if-cached` - Fetch uses any response in the HTTP cache matching the request, not paying attention to staleness. If there was no response, it returns a network error. (Can only be used when request’s mode is "same-origin". Any cached redirects will be followed assuming request’s redirect mode is "follow" and the redirects do not violate request’s mode.)
(Note: option descriptions are taken from https://fetch.spec.whatwg.org/#http-network-or-cache-fetch)
##### Example
```javascript
const fetch = require('make-fetch-happen').defaults({
cachePath: './my-cache'
})
// Will error with ENOTCACHED if we haven't already cached this url
fetch('https://registry.npmjs.org/make-fetch-happen', {
cache: 'only-if-cached'
})
// Will refresh any local content and cache the new response
fetch('https://registry.npmjs.org/make-fetch-happen', {
cache: 'reload'
})
// Will use any local data, even if stale. Otherwise, will hit network.
fetch('https://registry.npmjs.org/make-fetch-happen', {
cache: 'force-cache'
})
```
#### <a name="opts-proxy"></a> `> opts.proxy`
A string or `new url.URL()`-d URI to proxy through. Different Proxy handlers will be
used depending on the proxy's protocol.
Additionally, `process.env.HTTP_PROXY`, `process.env.HTTPS_PROXY`, and
`process.env.PROXY` are used if present and no `opts.proxy` value is provided.
(Pending) `process.env.NO_PROXY` may also be configured to skip proxying requests for all, or specific domains.
##### Example
```javascript
fetch('https://registry.npmjs.org/make-fetch-happen', {
proxy: 'https://corporate.yourcompany.proxy:4445'
})
fetch('https://registry.npmjs.org/make-fetch-happen', {
proxy: {
protocol: 'https:',
hostname: 'corporate.yourcompany.proxy',
port: 4445
}
})
```
#### <a name="opts-no-proxy"></a> `> opts.noProxy`
If present, should be a comma-separated string or an array of domain extensions
that a proxy should _not_ be used for.
This option may also be provided through `process.env.NO_PROXY`.
#### <a name="https-opts"></a> `> opts.ca, opts.cert, opts.key, opts.strictSSL`
These values are passed in directly to the HTTPS agent and will be used for both
proxied and unproxied outgoing HTTPS requests. They mostly correspond to the
same options the `https` module accepts, which will be themselves passed to
`tls.connect()`. `opts.strictSSL` corresponds to `rejectUnauthorized`.
#### <a name="opts-local-address"></a> `> opts.localAddress`
Passed directly to `http` and `https` request calls. Determines the local
address to bind to.
#### <a name="opts-max-sockets"></a> `> opts.maxSockets`
Default: 15
Maximum number of active concurrent sockets to use for the underlying
Http/Https/Proxy agents. This setting applies once per spawned agent.
15 is probably a _pretty good value_ for most use-cases, and balances speed
with, uh, not knocking out people's routers. 🤓
#### <a name="opts-retry"></a> `> opts.retry`
An object that can be used to tune request retry settings. Retries will only be attempted on the following conditions:
* Request method is NOT `POST` AND
* Request status is one of: `408`, `420`, `429`, or any status in the 500-range. OR
* Request errored with `ECONNRESET`, `ECONNREFUSED`, `EADDRINUSE`, `ETIMEDOUT`, or the `fetch` error `request-timeout`.
The following are worth noting as explicitly not retried:
* `getaddrinfo ENOTFOUND` and will be assumed to be either an unreachable domain or the user will be assumed offline. If a response is cached, it will be returned immediately.
If `opts.retry` is `false`, it is equivalent to `{retries: 0}`
If `opts.retry` is a number, it is equivalent to `{retries: num}`
The following retry options are available if you want more control over it:
* retries
* factor
* minTimeout
* maxTimeout
* randomize
For details on what each of these do, refer to the [`retry`](https://npm.im/retry) documentation.
##### Example
```javascript
fetch('https://flaky.site.com', {
retry: {
retries: 10,
randomize: true
}
})
fetch('http://reliable.site.com', {
retry: false
})
fetch('http://one-more.site.com', {
retry: 3
})
```
#### <a name="opts-onretry"></a> `> opts.onRetry`
A function called whenever a retry is attempted.
##### Example
```javascript
fetch('https://flaky.site.com', {
onRetry() {
console.log('we will retry!')
}
})
```
#### <a name="opts-integrity"></a> `> opts.integrity`
Matches the response body against the given [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) metadata. If verification fails, the request will fail with an `EINTEGRITY` error.
`integrity` may either be a string or an [`ssri`](https://npm.im/ssri) `Integrity`-like.
##### Example
```javascript
fetch('https://registry.npmjs.org/make-fetch-happen/-/make-fetch-happen-1.0.0.tgz', {
integrity: 'sha1-o47j7zAYnedYFn1dF/fR9OV3z8Q='
}) // -> ok
fetch('https://malicious-registry.org/make-fetch-happen/-/make-fetch-happen-1.0.0.tgz', {
integrity: 'sha1-o47j7zAYnedYFn1dF/fR9OV3z8Q='
}) // Error: EINTEGRITY
```
#### <a name="opts-dns"></a> `> opts.dns`
An object that provides options for the built-in DNS cache. The following options are available:
Note: Due to limitations in the current proxy agent implementation, users of proxies will not benefit from the DNS cache.
* `ttl`: Milliseconds to keep cached DNS responses for. Defaults to `5 * 60 * 1000` (5 minutes)
* `lookup`: A custom lookup function, see [`dns.lookup()`](https://nodejs.org/api/dns.html#dnslookuphostname-options-callback) for implementation details. Defaults to `require('dns').lookup`.
+214
View File
@@ -0,0 +1,214 @@
'use strict'
const LRU = require('lru-cache')
const url = require('url')
const isLambda = require('is-lambda')
const dns = require('./dns.js')
const AGENT_CACHE = new LRU({ max: 50 })
const HttpAgent = require('agentkeepalive')
const HttpsAgent = HttpAgent.HttpsAgent
module.exports = getAgent
const getAgentTimeout = timeout =>
typeof timeout !== 'number' || !timeout ? 0 : timeout + 1
const getMaxSockets = maxSockets => maxSockets || 15
function getAgent (uri, opts) {
const parsedUri = new url.URL(typeof uri === 'string' ? uri : uri.url)
const isHttps = parsedUri.protocol === 'https:'
const pxuri = getProxyUri(parsedUri.href, opts)
// If opts.timeout is zero, set the agentTimeout to zero as well. A timeout
// of zero disables the timeout behavior (OS limits still apply). Else, if
// opts.timeout is a non-zero value, set it to timeout + 1, to ensure that
// the node-fetch-npm timeout will always fire first, giving us more
// consistent errors.
const agentTimeout = getAgentTimeout(opts.timeout)
const agentMaxSockets = getMaxSockets(opts.maxSockets)
const key = [
`https:${isHttps}`,
pxuri
? `proxy:${pxuri.protocol}//${pxuri.host}:${pxuri.port}`
: '>no-proxy<',
`local-address:${opts.localAddress || '>no-local-address<'}`,
`strict-ssl:${isHttps ? opts.rejectUnauthorized : '>no-strict-ssl<'}`,
`ca:${(isHttps && opts.ca) || '>no-ca<'}`,
`cert:${(isHttps && opts.cert) || '>no-cert<'}`,
`key:${(isHttps && opts.key) || '>no-key<'}`,
`timeout:${agentTimeout}`,
`maxSockets:${agentMaxSockets}`,
].join(':')
if (opts.agent != null) { // `agent: false` has special behavior!
return opts.agent
}
// keep alive in AWS lambda makes no sense
const lambdaAgent = !isLambda ? null
: isHttps ? require('https').globalAgent
: require('http').globalAgent
if (isLambda && !pxuri) {
return lambdaAgent
}
if (AGENT_CACHE.peek(key)) {
return AGENT_CACHE.get(key)
}
if (pxuri) {
const pxopts = isLambda ? {
...opts,
agent: lambdaAgent,
} : opts
const proxy = getProxy(pxuri, pxopts, isHttps)
AGENT_CACHE.set(key, proxy)
return proxy
}
const agent = isHttps ? new HttpsAgent({
maxSockets: agentMaxSockets,
ca: opts.ca,
cert: opts.cert,
key: opts.key,
localAddress: opts.localAddress,
rejectUnauthorized: opts.rejectUnauthorized,
timeout: agentTimeout,
freeSocketTimeout: 15000,
lookup: dns.getLookup(opts.dns),
}) : new HttpAgent({
maxSockets: agentMaxSockets,
localAddress: opts.localAddress,
timeout: agentTimeout,
freeSocketTimeout: 15000,
lookup: dns.getLookup(opts.dns),
})
AGENT_CACHE.set(key, agent)
return agent
}
function checkNoProxy (uri, opts) {
const host = new url.URL(uri).hostname.split('.').reverse()
let noproxy = (opts.noProxy || getProcessEnv('no_proxy'))
if (typeof noproxy === 'string') {
noproxy = noproxy.split(',').map(n => n.trim())
}
return noproxy && noproxy.some(no => {
const noParts = no.split('.').filter(x => x).reverse()
if (!noParts.length) {
return false
}
for (let i = 0; i < noParts.length; i++) {
if (host[i] !== noParts[i]) {
return false
}
}
return true
})
}
module.exports.getProcessEnv = getProcessEnv
function getProcessEnv (env) {
if (!env) {
return
}
let value
if (Array.isArray(env)) {
for (const e of env) {
value = process.env[e] ||
process.env[e.toUpperCase()] ||
process.env[e.toLowerCase()]
if (typeof value !== 'undefined') {
break
}
}
}
if (typeof env === 'string') {
value = process.env[env] ||
process.env[env.toUpperCase()] ||
process.env[env.toLowerCase()]
}
return value
}
module.exports.getProxyUri = getProxyUri
function getProxyUri (uri, opts) {
const protocol = new url.URL(uri).protocol
const proxy = opts.proxy ||
(
protocol === 'https:' &&
getProcessEnv('https_proxy')
) ||
(
protocol === 'http:' &&
getProcessEnv(['https_proxy', 'http_proxy', 'proxy'])
)
if (!proxy) {
return null
}
const parsedProxy = (typeof proxy === 'string') ? new url.URL(proxy) : proxy
return !checkNoProxy(uri, opts) && parsedProxy
}
const getAuth = u =>
u.username && u.password ? decodeURIComponent(`${u.username}:${u.password}`)
: u.username ? decodeURIComponent(u.username)
: null
const getPath = u => u.pathname + u.search + u.hash
const HttpProxyAgent = require('http-proxy-agent')
const HttpsProxyAgent = require('https-proxy-agent')
const { SocksProxyAgent } = require('socks-proxy-agent')
module.exports.getProxy = getProxy
function getProxy (proxyUrl, opts, isHttps) {
// our current proxy agents do not support an overridden dns lookup method, so will not
// benefit from the dns cache
const popts = {
host: proxyUrl.hostname,
port: proxyUrl.port,
protocol: proxyUrl.protocol,
path: getPath(proxyUrl),
auth: getAuth(proxyUrl),
ca: opts.ca,
cert: opts.cert,
key: opts.key,
timeout: getAgentTimeout(opts.timeout),
localAddress: opts.localAddress,
maxSockets: getMaxSockets(opts.maxSockets),
rejectUnauthorized: opts.rejectUnauthorized,
}
if (proxyUrl.protocol === 'http:' || proxyUrl.protocol === 'https:') {
if (!isHttps) {
return new HttpProxyAgent(popts)
} else {
return new HttpsProxyAgent(popts)
}
} else if (proxyUrl.protocol.startsWith('socks')) {
// socks-proxy-agent uses hostname not host
popts.hostname = popts.host
delete popts.host
return new SocksProxyAgent(popts)
} else {
throw Object.assign(
new Error(`unsupported proxy protocol: '${proxyUrl.protocol}'`),
{
code: 'EUNSUPPORTEDPROXY',
url: proxyUrl.href,
}
)
}
}
+444
View File
@@ -0,0 +1,444 @@
const { Request, Response } = require('minipass-fetch')
const Minipass = require('minipass')
const MinipassFlush = require('minipass-flush')
const cacache = require('cacache')
const url = require('url')
const CachingMinipassPipeline = require('../pipeline.js')
const CachePolicy = require('./policy.js')
const cacheKey = require('./key.js')
const remote = require('../remote.js')
const hasOwnProperty = (obj, prop) => Object.prototype.hasOwnProperty.call(obj, prop)
// allow list for request headers that will be written to the cache index
// note: we will also store any request headers
// that are named in a response's vary header
const KEEP_REQUEST_HEADERS = [
'accept-charset',
'accept-encoding',
'accept-language',
'accept',
'cache-control',
]
// allow list for response headers that will be written to the cache index
// note: we must not store the real response's age header, or when we load
// a cache policy based on the metadata it will think the cached response
// is always stale
const KEEP_RESPONSE_HEADERS = [
'cache-control',
'content-encoding',
'content-language',
'content-type',
'date',
'etag',
'expires',
'last-modified',
'link',
'location',
'pragma',
'vary',
]
// return an object containing all metadata to be written to the index
const getMetadata = (request, response, options) => {
const metadata = {
time: Date.now(),
url: request.url,
reqHeaders: {},
resHeaders: {},
// options on which we must match the request and vary the response
options: {
compress: options.compress != null ? options.compress : request.compress,
},
}
// only save the status if it's not a 200 or 304
if (response.status !== 200 && response.status !== 304) {
metadata.status = response.status
}
for (const name of KEEP_REQUEST_HEADERS) {
if (request.headers.has(name)) {
metadata.reqHeaders[name] = request.headers.get(name)
}
}
// if the request's host header differs from the host in the url
// we need to keep it, otherwise it's just noise and we ignore it
const host = request.headers.get('host')
const parsedUrl = new url.URL(request.url)
if (host && parsedUrl.host !== host) {
metadata.reqHeaders.host = host
}
// if the response has a vary header, make sure
// we store the relevant request headers too
if (response.headers.has('vary')) {
const vary = response.headers.get('vary')
// a vary of "*" means every header causes a different response.
// in that scenario, we do not include any additional headers
// as the freshness check will always fail anyway and we don't
// want to bloat the cache indexes
if (vary !== '*') {
// copy any other request headers that will vary the response
const varyHeaders = vary.trim().toLowerCase().split(/\s*,\s*/)
for (const name of varyHeaders) {
if (request.headers.has(name)) {
metadata.reqHeaders[name] = request.headers.get(name)
}
}
}
}
for (const name of KEEP_RESPONSE_HEADERS) {
if (response.headers.has(name)) {
metadata.resHeaders[name] = response.headers.get(name)
}
}
return metadata
}
// symbols used to hide objects that may be lazily evaluated in a getter
const _request = Symbol('request')
const _response = Symbol('response')
const _policy = Symbol('policy')
class CacheEntry {
constructor ({ entry, request, response, options }) {
if (entry) {
this.key = entry.key
this.entry = entry
// previous versions of this module didn't write an explicit timestamp in
// the metadata, so fall back to the entry's timestamp. we can't use the
// entry timestamp to determine staleness because cacache will update it
// when it verifies its data
this.entry.metadata.time = this.entry.metadata.time || this.entry.time
} else {
this.key = cacheKey(request)
}
this.options = options
// these properties are behind getters that lazily evaluate
this[_request] = request
this[_response] = response
this[_policy] = null
}
// returns a CacheEntry instance that satisfies the given request
// or undefined if no existing entry satisfies
static async find (request, options) {
try {
// compacts the index and returns an array of unique entries
var matches = await cacache.index.compact(options.cachePath, cacheKey(request), (A, B) => {
const entryA = new CacheEntry({ entry: A, options })
const entryB = new CacheEntry({ entry: B, options })
return entryA.policy.satisfies(entryB.request)
}, {
validateEntry: (entry) => {
// clean out entries with a buggy content-encoding value
if (entry.metadata &&
entry.metadata.resHeaders &&
entry.metadata.resHeaders['content-encoding'] === null) {
return false
}
// if an integrity is null, it needs to have a status specified
if (entry.integrity === null) {
return !!(entry.metadata && entry.metadata.status)
}
return true
},
})
} catch (err) {
// if the compact request fails, ignore the error and return
return
}
// a cache mode of 'reload' means to behave as though we have no cache
// on the way to the network. return undefined to allow cacheFetch to
// create a brand new request no matter what.
if (options.cache === 'reload') {
return
}
// find the specific entry that satisfies the request
let match
for (const entry of matches) {
const _entry = new CacheEntry({
entry,
options,
})
if (_entry.policy.satisfies(request)) {
match = _entry
break
}
}
return match
}
// if the user made a PUT/POST/PATCH then we invalidate our
// cache for the same url by deleting the index entirely
static async invalidate (request, options) {
const key = cacheKey(request)
try {
await cacache.rm.entry(options.cachePath, key, { removeFully: true })
} catch (err) {
// ignore errors
}
}
get request () {
if (!this[_request]) {
this[_request] = new Request(this.entry.metadata.url, {
method: 'GET',
headers: this.entry.metadata.reqHeaders,
...this.entry.metadata.options,
})
}
return this[_request]
}
get response () {
if (!this[_response]) {
this[_response] = new Response(null, {
url: this.entry.metadata.url,
counter: this.options.counter,
status: this.entry.metadata.status || 200,
headers: {
...this.entry.metadata.resHeaders,
'content-length': this.entry.size,
},
})
}
return this[_response]
}
get policy () {
if (!this[_policy]) {
this[_policy] = new CachePolicy({
entry: this.entry,
request: this.request,
response: this.response,
options: this.options,
})
}
return this[_policy]
}
// wraps the response in a pipeline that stores the data
// in the cache while the user consumes it
async store (status) {
// if we got a status other than 200, 301, or 308,
// or the CachePolicy forbid storage, append the
// cache status header and return it untouched
if (
this.request.method !== 'GET' ||
![200, 301, 308].includes(this.response.status) ||
!this.policy.storable()
) {
this.response.headers.set('x-local-cache-status', 'skip')
return this.response
}
const size = this.response.headers.get('content-length')
const cacheOpts = {
algorithms: this.options.algorithms,
metadata: getMetadata(this.request, this.response, this.options),
size,
integrity: this.options.integrity,
integrityEmitter: this.response.body.hasIntegrityEmitter && this.response.body,
}
let body = null
// we only set a body if the status is a 200, redirects are
// stored as metadata only
if (this.response.status === 200) {
let cacheWriteResolve, cacheWriteReject
const cacheWritePromise = new Promise((resolve, reject) => {
cacheWriteResolve = resolve
cacheWriteReject = reject
})
body = new CachingMinipassPipeline({ events: ['integrity', 'size'] }, new MinipassFlush({
flush () {
return cacheWritePromise
},
}))
// this is always true since if we aren't reusing the one from the remote fetch, we
// are using the one from cacache
body.hasIntegrityEmitter = true
const onResume = () => {
const tee = new Minipass()
const cacheStream = cacache.put.stream(this.options.cachePath, this.key, cacheOpts)
// re-emit the integrity and size events on our new response body so they can be reused
cacheStream.on('integrity', i => body.emit('integrity', i))
cacheStream.on('size', s => body.emit('size', s))
// stick a flag on here so downstream users will know if they can expect integrity events
tee.pipe(cacheStream)
// TODO if the cache write fails, log a warning but return the response anyway
// eslint-disable-next-line promise/catch-or-return
cacheStream.promise().then(cacheWriteResolve, cacheWriteReject)
body.unshift(tee)
body.unshift(this.response.body)
}
body.once('resume', onResume)
body.once('end', () => body.removeListener('resume', onResume))
} else {
await cacache.index.insert(this.options.cachePath, this.key, null, cacheOpts)
}
// note: we do not set the x-local-cache-hash header because we do not know
// the hash value until after the write to the cache completes, which doesn't
// happen until after the response has been sent and it's too late to write
// the header anyway
this.response.headers.set('x-local-cache', encodeURIComponent(this.options.cachePath))
this.response.headers.set('x-local-cache-key', encodeURIComponent(this.key))
this.response.headers.set('x-local-cache-mode', 'stream')
this.response.headers.set('x-local-cache-status', status)
this.response.headers.set('x-local-cache-time', new Date().toISOString())
const newResponse = new Response(body, {
url: this.response.url,
status: this.response.status,
headers: this.response.headers,
counter: this.options.counter,
})
return newResponse
}
// use the cached data to create a response and return it
async respond (method, options, status) {
let response
if (method === 'HEAD' || [301, 308].includes(this.response.status)) {
// if the request is a HEAD, or the response is a redirect,
// then the metadata in the entry already includes everything
// we need to build a response
response = this.response
} else {
// we're responding with a full cached response, so create a body
// that reads from cacache and attach it to a new Response
const body = new Minipass()
const headers = { ...this.policy.responseHeaders() }
const onResume = () => {
const cacheStream = cacache.get.stream.byDigest(
this.options.cachePath, this.entry.integrity, { memoize: this.options.memoize }
)
cacheStream.on('error', async (err) => {
cacheStream.pause()
if (err.code === 'EINTEGRITY') {
await cacache.rm.content(
this.options.cachePath, this.entry.integrity, { memoize: this.options.memoize }
)
}
if (err.code === 'ENOENT' || err.code === 'EINTEGRITY') {
await CacheEntry.invalidate(this.request, this.options)
}
body.emit('error', err)
cacheStream.resume()
})
// emit the integrity and size events based on our metadata so we're consistent
body.emit('integrity', this.entry.integrity)
body.emit('size', Number(headers['content-length']))
cacheStream.pipe(body)
}
body.once('resume', onResume)
body.once('end', () => body.removeListener('resume', onResume))
response = new Response(body, {
url: this.entry.metadata.url,
counter: options.counter,
status: 200,
headers,
})
}
response.headers.set('x-local-cache', encodeURIComponent(this.options.cachePath))
response.headers.set('x-local-cache-hash', encodeURIComponent(this.entry.integrity))
response.headers.set('x-local-cache-key', encodeURIComponent(this.key))
response.headers.set('x-local-cache-mode', 'stream')
response.headers.set('x-local-cache-status', status)
response.headers.set('x-local-cache-time', new Date(this.entry.metadata.time).toUTCString())
return response
}
// use the provided request along with this cache entry to
// revalidate the stored response. returns a response, either
// from the cache or from the update
async revalidate (request, options) {
const revalidateRequest = new Request(request, {
headers: this.policy.revalidationHeaders(request),
})
try {
// NOTE: be sure to remove the headers property from the
// user supplied options, since we have already defined
// them on the new request object. if they're still in the
// options then those will overwrite the ones from the policy
var response = await remote(revalidateRequest, {
...options,
headers: undefined,
})
} catch (err) {
// if the network fetch fails, return the stale
// cached response unless it has a cache-control
// of 'must-revalidate'
if (!this.policy.mustRevalidate) {
return this.respond(request.method, options, 'stale')
}
throw err
}
if (this.policy.revalidated(revalidateRequest, response)) {
// we got a 304, write a new index to the cache and respond from cache
const metadata = getMetadata(request, response, options)
// 304 responses do not include headers that are specific to the response data
// since they do not include a body, so we copy values for headers that were
// in the old cache entry to the new one, if the new metadata does not already
// include that header
for (const name of KEEP_RESPONSE_HEADERS) {
if (
!hasOwnProperty(metadata.resHeaders, name) &&
hasOwnProperty(this.entry.metadata.resHeaders, name)
) {
metadata.resHeaders[name] = this.entry.metadata.resHeaders[name]
}
}
try {
await cacache.index.insert(options.cachePath, this.key, this.entry.integrity, {
size: this.entry.size,
metadata,
})
} catch (err) {
// if updating the cache index fails, we ignore it and
// respond anyway
}
return this.respond(request.method, options, 'revalidated')
}
// if we got a modified response, create a new entry based on it
const newEntry = new CacheEntry({
request,
response,
options,
})
// respond with the new entry while writing it to the cache
return newEntry.store('updated')
}
}
module.exports = CacheEntry
+11
View File
@@ -0,0 +1,11 @@
class NotCachedError extends Error {
constructor (url) {
/* eslint-disable-next-line max-len */
super(`request to ${url} failed: cache mode is 'only-if-cached' but no cached response is available.`)
this.code = 'ENOTCACHED'
}
}
module.exports = {
NotCachedError,
}
+49
View File
@@ -0,0 +1,49 @@
const { NotCachedError } = require('./errors.js')
const CacheEntry = require('./entry.js')
const remote = require('../remote.js')
// do whatever is necessary to get a Response and return it
const cacheFetch = async (request, options) => {
// try to find a cached entry that satisfies this request
const entry = await CacheEntry.find(request, options)
if (!entry) {
// no cached result, if the cache mode is 'only-if-cached' that's a failure
if (options.cache === 'only-if-cached') {
throw new NotCachedError(request.url)
}
// otherwise, we make a request, store it and return it
const response = await remote(request, options)
const newEntry = new CacheEntry({ request, response, options })
return newEntry.store('miss')
}
// we have a cached response that satisfies this request, however if the cache
// mode is 'no-cache' then we send the revalidation request no matter what
if (options.cache === 'no-cache') {
return entry.revalidate(request, options)
}
// if the cached entry is not stale, or if the cache mode is 'force-cache' or
// 'only-if-cached' we can respond with the cached entry. set the status
// based on the result of needsRevalidation and respond
const _needsRevalidation = entry.policy.needsRevalidation(request)
if (options.cache === 'force-cache' ||
options.cache === 'only-if-cached' ||
!_needsRevalidation) {
return entry.respond(request.method, options, _needsRevalidation ? 'stale' : 'hit')
}
// if we got here, the cache entry is stale so revalidate it
return entry.revalidate(request, options)
}
cacheFetch.invalidate = async (request, options) => {
if (!options.cachePath) {
return
}
return CacheEntry.invalidate(request, options)
}
module.exports = cacheFetch
+17
View File
@@ -0,0 +1,17 @@
const { URL, format } = require('url')
// options passed to url.format() when generating a key
const formatOptions = {
auth: false,
fragment: false,
search: true,
unicode: false,
}
// returns a string to be used as the cache key for the Request
const cacheKey = (request) => {
const parsed = new URL(request.url)
return `make-fetch-happen:request-cache:${format(parsed, formatOptions)}`
}
module.exports = cacheKey
+161
View File
@@ -0,0 +1,161 @@
const CacheSemantics = require('http-cache-semantics')
const Negotiator = require('negotiator')
const ssri = require('ssri')
// options passed to http-cache-semantics constructor
const policyOptions = {
shared: false,
ignoreCargoCult: true,
}
// a fake empty response, used when only testing the
// request for storability
const emptyResponse = { status: 200, headers: {} }
// returns a plain object representation of the Request
const requestObject = (request) => {
const _obj = {
method: request.method,
url: request.url,
headers: {},
compress: request.compress,
}
request.headers.forEach((value, key) => {
_obj.headers[key] = value
})
return _obj
}
// returns a plain object representation of the Response
const responseObject = (response) => {
const _obj = {
status: response.status,
headers: {},
}
response.headers.forEach((value, key) => {
_obj.headers[key] = value
})
return _obj
}
class CachePolicy {
constructor ({ entry, request, response, options }) {
this.entry = entry
this.request = requestObject(request)
this.response = responseObject(response)
this.options = options
this.policy = new CacheSemantics(this.request, this.response, policyOptions)
if (this.entry) {
// if we have an entry, copy the timestamp to the _responseTime
// this is necessary because the CacheSemantics constructor forces
// the value to Date.now() which means a policy created from a
// cache entry is likely to always identify itself as stale
this.policy._responseTime = this.entry.metadata.time
}
}
// static method to quickly determine if a request alone is storable
static storable (request, options) {
// no cachePath means no caching
if (!options.cachePath) {
return false
}
// user explicitly asked not to cache
if (options.cache === 'no-store') {
return false
}
// we only cache GET and HEAD requests
if (!['GET', 'HEAD'].includes(request.method)) {
return false
}
// otherwise, let http-cache-semantics make the decision
// based on the request's headers
const policy = new CacheSemantics(requestObject(request), emptyResponse, policyOptions)
return policy.storable()
}
// returns true if the policy satisfies the request
satisfies (request) {
const _req = requestObject(request)
if (this.request.headers.host !== _req.headers.host) {
return false
}
if (this.request.compress !== _req.compress) {
return false
}
const negotiatorA = new Negotiator(this.request)
const negotiatorB = new Negotiator(_req)
if (JSON.stringify(negotiatorA.mediaTypes()) !== JSON.stringify(negotiatorB.mediaTypes())) {
return false
}
if (JSON.stringify(negotiatorA.languages()) !== JSON.stringify(negotiatorB.languages())) {
return false
}
if (JSON.stringify(negotiatorA.encodings()) !== JSON.stringify(negotiatorB.encodings())) {
return false
}
if (this.options.integrity) {
return ssri.parse(this.options.integrity).match(this.entry.integrity)
}
return true
}
// returns true if the request and response allow caching
storable () {
return this.policy.storable()
}
// NOTE: this is a hack to avoid parsing the cache-control
// header ourselves, it returns true if the response's
// cache-control contains must-revalidate
get mustRevalidate () {
return !!this.policy._rescc['must-revalidate']
}
// returns true if the cached response requires revalidation
// for the given request
needsRevalidation (request) {
const _req = requestObject(request)
// force method to GET because we only cache GETs
// but can serve a HEAD from a cached GET
_req.method = 'GET'
return !this.policy.satisfiesWithoutRevalidation(_req)
}
responseHeaders () {
return this.policy.responseHeaders()
}
// returns a new object containing the appropriate headers
// to send a revalidation request
revalidationHeaders (request) {
const _req = requestObject(request)
return this.policy.revalidationHeaders(_req)
}
// returns true if the request/response was revalidated
// successfully. returns false if a new response was received
revalidated (request, response) {
const _req = requestObject(request)
const _res = responseObject(response)
const policy = this.policy.revalidatedPolicy(_req, _res)
return !policy.modified
}
}
module.exports = CachePolicy
+49
View File
@@ -0,0 +1,49 @@
const LRUCache = require('lru-cache')
const dns = require('dns')
const defaultOptions = exports.defaultOptions = {
family: undefined,
hints: dns.ADDRCONFIG,
all: false,
verbatim: undefined,
}
const lookupCache = exports.lookupCache = new LRUCache({ max: 50 })
// this is a factory so that each request can have its own opts (i.e. ttl)
// while still sharing the cache across all requests
exports.getLookup = (dnsOptions) => {
return (hostname, options, callback) => {
if (typeof options === 'function') {
callback = options
options = null
} else if (typeof options === 'number') {
options = { family: options }
}
options = { ...defaultOptions, ...options }
const key = JSON.stringify({
hostname,
family: options.family,
hints: options.hints,
all: options.all,
verbatim: options.verbatim,
})
if (lookupCache.has(key)) {
const [address, family] = lookupCache.get(key)
process.nextTick(callback, null, address, family)
return
}
dnsOptions.lookup(hostname, options, (err, address, family) => {
if (err) {
return callback(err)
}
lookupCache.set(key, [address, family], { ttl: dnsOptions.ttl })
return callback(null, address, family)
})
}
}
+118
View File
@@ -0,0 +1,118 @@
'use strict'
const { FetchError, Request, isRedirect } = require('minipass-fetch')
const url = require('url')
const CachePolicy = require('./cache/policy.js')
const cache = require('./cache/index.js')
const remote = require('./remote.js')
// given a Request, a Response and user options
// return true if the response is a redirect that
// can be followed. we throw errors that will result
// in the fetch being rejected if the redirect is
// possible but invalid for some reason
const canFollowRedirect = (request, response, options) => {
if (!isRedirect(response.status)) {
return false
}
if (options.redirect === 'manual') {
return false
}
if (options.redirect === 'error') {
throw new FetchError(`redirect mode is set to error: ${request.url}`,
'no-redirect', { code: 'ENOREDIRECT' })
}
if (!response.headers.has('location')) {
throw new FetchError(`redirect location header missing for: ${request.url}`,
'no-location', { code: 'EINVALIDREDIRECT' })
}
if (request.counter >= request.follow) {
throw new FetchError(`maximum redirect reached at: ${request.url}`,
'max-redirect', { code: 'EMAXREDIRECT' })
}
return true
}
// given a Request, a Response, and the user's options return an object
// with a new Request and a new options object that will be used for
// following the redirect
const getRedirect = (request, response, options) => {
const _opts = { ...options }
const location = response.headers.get('location')
const redirectUrl = new url.URL(location, /^https?:/.test(location) ? undefined : request.url)
// Comment below is used under the following license:
/**
* @license
* Copyright (c) 2010-2012 Mikeal Rogers
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an "AS
* IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language
* governing permissions and limitations under the License.
*/
// Remove authorization if changing hostnames (but not if just
// changing ports or protocols). This matches the behavior of request:
// https://github.com/request/request/blob/b12a6245/lib/redirect.js#L134-L138
if (new url.URL(request.url).hostname !== redirectUrl.hostname) {
request.headers.delete('authorization')
request.headers.delete('cookie')
}
// for POST request with 301/302 response, or any request with 303 response,
// use GET when following redirect
if (
response.status === 303 ||
(request.method === 'POST' && [301, 302].includes(response.status))
) {
_opts.method = 'GET'
_opts.body = null
request.headers.delete('content-length')
}
_opts.headers = {}
request.headers.forEach((value, key) => {
_opts.headers[key] = value
})
_opts.counter = ++request.counter
const redirectReq = new Request(url.format(redirectUrl), _opts)
return {
request: redirectReq,
options: _opts,
}
}
const fetch = async (request, options) => {
const response = CachePolicy.storable(request, options)
? await cache(request, options)
: await remote(request, options)
// if the request wasn't a GET or HEAD, and the response
// status is between 200 and 399 inclusive, invalidate the
// request url
if (!['GET', 'HEAD'].includes(request.method) &&
response.status >= 200 &&
response.status <= 399) {
await cache.invalidate(request, options)
}
if (!canFollowRedirect(request, response, options)) {
return response
}
const redirect = getRedirect(request, response, options)
return fetch(redirect.request, redirect.options)
}
module.exports = fetch
+41
View File
@@ -0,0 +1,41 @@
const { FetchError, Headers, Request, Response } = require('minipass-fetch')
const configureOptions = require('./options.js')
const fetch = require('./fetch.js')
const makeFetchHappen = (url, opts) => {
const options = configureOptions(opts)
const request = new Request(url, options)
return fetch(request, options)
}
makeFetchHappen.defaults = (defaultUrl, defaultOptions = {}, wrappedFetch = makeFetchHappen) => {
if (typeof defaultUrl === 'object') {
defaultOptions = defaultUrl
defaultUrl = null
}
const defaultedFetch = (url, options = {}) => {
const finalUrl = url || defaultUrl
const finalOptions = {
...defaultOptions,
...options,
headers: {
...defaultOptions.headers,
...options.headers,
},
}
return wrappedFetch(finalUrl, finalOptions)
}
defaultedFetch.defaults = (defaultUrl1, defaultOptions1 = {}) =>
makeFetchHappen.defaults(defaultUrl1, defaultOptions1, defaultedFetch)
return defaultedFetch
}
module.exports = makeFetchHappen
module.exports.FetchError = FetchError
module.exports.Headers = Headers
module.exports.Request = Request
module.exports.Response = Response
+52
View File
@@ -0,0 +1,52 @@
const dns = require('dns')
const conditionalHeaders = [
'if-modified-since',
'if-none-match',
'if-unmodified-since',
'if-match',
'if-range',
]
const configureOptions = (opts) => {
const { strictSSL, ...options } = { ...opts }
options.method = options.method ? options.method.toUpperCase() : 'GET'
options.rejectUnauthorized = strictSSL !== false
if (!options.retry) {
options.retry = { retries: 0 }
} else if (typeof options.retry === 'string') {
const retries = parseInt(options.retry, 10)
if (isFinite(retries)) {
options.retry = { retries }
} else {
options.retry = { retries: 0 }
}
} else if (typeof options.retry === 'number') {
options.retry = { retries: options.retry }
} else {
options.retry = { retries: 0, ...options.retry }
}
options.dns = { ttl: 5 * 60 * 1000, lookup: dns.lookup, ...options.dns }
options.cache = options.cache || 'default'
if (options.cache === 'default') {
const hasConditionalHeader = Object.keys(options.headers || {}).some((name) => {
return conditionalHeaders.includes(name.toLowerCase())
})
if (hasConditionalHeader) {
options.cache = 'no-store'
}
}
// cacheManager is deprecated, but if it's set and
// cachePath is not we should copy it to the new field
if (options.cacheManager && !options.cachePath) {
options.cachePath = options.cacheManager
}
return options
}
module.exports = configureOptions
+41
View File
@@ -0,0 +1,41 @@
'use strict'
const MinipassPipeline = require('minipass-pipeline')
class CachingMinipassPipeline extends MinipassPipeline {
#events = []
#data = new Map()
constructor (opts, ...streams) {
// CRITICAL: do NOT pass the streams to the call to super(), this will start
// the flow of data and potentially cause the events we need to catch to emit
// before we've finished our own setup. instead we call super() with no args,
// finish our setup, and then push the streams into ourselves to start the
// data flow
super()
this.#events = opts.events
/* istanbul ignore next - coverage disabled because this is pointless to test here */
if (streams.length) {
this.push(...streams)
}
}
on (event, handler) {
if (this.#events.includes(event) && this.#data.has(event)) {
return handler(...this.#data.get(event))
}
return super.on(event, handler)
}
emit (event, ...data) {
if (this.#events.includes(event)) {
this.#data.set(event, data)
}
return super.emit(event, ...data)
}
}
module.exports = CachingMinipassPipeline
+121
View File
@@ -0,0 +1,121 @@
const Minipass = require('minipass')
const fetch = require('minipass-fetch')
const promiseRetry = require('promise-retry')
const ssri = require('ssri')
const CachingMinipassPipeline = require('./pipeline.js')
const getAgent = require('./agent.js')
const pkg = require('../package.json')
const USER_AGENT = `${pkg.name}/${pkg.version} (+https://npm.im/${pkg.name})`
const RETRY_ERRORS = [
'ECONNRESET', // remote socket closed on us
'ECONNREFUSED', // remote host refused to open connection
'EADDRINUSE', // failed to bind to a local port (proxy?)
'ETIMEDOUT', // someone in the transaction is WAY TOO SLOW
'ERR_SOCKET_TIMEOUT', // same as above, but this one comes from agentkeepalive
// Known codes we do NOT retry on:
// ENOTFOUND (getaddrinfo failure. Either bad hostname, or offline)
]
const RETRY_TYPES = [
'request-timeout',
]
// make a request directly to the remote source,
// retrying certain classes of errors as well as
// following redirects (through the cache if necessary)
// and verifying response integrity
const remoteFetch = (request, options) => {
const agent = getAgent(request.url, options)
if (!request.headers.has('connection')) {
request.headers.set('connection', agent ? 'keep-alive' : 'close')
}
if (!request.headers.has('user-agent')) {
request.headers.set('user-agent', USER_AGENT)
}
// keep our own options since we're overriding the agent
// and the redirect mode
const _opts = {
...options,
agent,
redirect: 'manual',
}
return promiseRetry(async (retryHandler, attemptNum) => {
const req = new fetch.Request(request, _opts)
try {
let res = await fetch(req, _opts)
if (_opts.integrity && res.status === 200) {
// we got a 200 response and the user has specified an expected
// integrity value, so wrap the response in an ssri stream to verify it
const integrityStream = ssri.integrityStream({
algorithms: _opts.algorithms,
integrity: _opts.integrity,
size: _opts.size,
})
const pipeline = new CachingMinipassPipeline({
events: ['integrity', 'size'],
}, res.body, integrityStream)
// we also propagate the integrity and size events out to the pipeline so we can use
// this new response body as an integrityEmitter for cacache
integrityStream.on('integrity', i => pipeline.emit('integrity', i))
integrityStream.on('size', s => pipeline.emit('size', s))
res = new fetch.Response(pipeline, res)
// set an explicit flag so we know if our response body will emit integrity and size
res.body.hasIntegrityEmitter = true
}
res.headers.set('x-fetch-attempts', attemptNum)
// do not retry POST requests, or requests with a streaming body
// do retry requests with a 408, 420, 429 or 500+ status in the response
const isStream = Minipass.isStream(req.body)
const isRetriable = req.method !== 'POST' &&
!isStream &&
([408, 420, 429].includes(res.status) || res.status >= 500)
if (isRetriable) {
if (typeof options.onRetry === 'function') {
options.onRetry(res)
}
return retryHandler(res)
}
return res
} catch (err) {
const code = (err.code === 'EPROMISERETRY')
? err.retried.code
: err.code
// err.retried will be the thing that was thrown from above
// if it's a response, we just got a bad status code and we
// can re-throw to allow the retry
const isRetryError = err.retried instanceof fetch.Response ||
(RETRY_ERRORS.includes(code) && RETRY_TYPES.includes(err.type))
if (req.method === 'POST' || isRetryError) {
throw err
}
if (typeof options.onRetry === 'function') {
options.onRetry(err)
}
return retryHandler(err)
}
}, options.retry).catch((err) => {
// don't reject for http errors, just return them
if (err.status >= 400 && err.type !== 'system') {
return err
}
throw err
})
}
module.exports = remoteFetch
@@ -0,0 +1,20 @@
<!-- This file is automatically added by @npmcli/template-oss. Do not edit. -->
ISC License
Copyright npm, Inc.
Permission to use, copy, modify, and/or distribute this
software for any purpose with or without fee is hereby
granted, provided that the above copyright notice and this
permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND NPM DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO
EVENT SHALL NPM BE LIABLE FOR ANY SPECIAL, DIRECT,
INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE
USE OR PERFORMANCE OF THIS SOFTWARE.
@@ -0,0 +1,62 @@
# @npmcli/fs
polyfills, and extensions, of the core `fs` module.
## Features
- all exposed functions return promises
- `fs.rm` polyfill for node versions < 14.14.0
- `fs.mkdir` polyfill adding support for the `recursive` and `force` options in node versions < 10.12.0
- `fs.copyFile` extended to accept an `owner` option
- `fs.mkdir` extended to accept an `owner` option
- `fs.mkdtemp` extended to accept an `owner` option
- `fs.writeFile` extended to accept an `owner` option
- `fs.withTempDir` added
- `fs.withOwner` added
- `fs.withOwnerSync` added
- `fs.cp` polyfill for node < 16.7.0
## The `owner` option
The `copyFile`, `mkdir`, `mkdtemp`, `writeFile`, and `withTempDir` functions
all accept a new `owner` property in their options. It can be used in two ways:
- `{ owner: { uid: 100, gid: 100 } }` - set the `uid` and `gid` explicitly
- `{ owner: 100 }` - use one value, will set both `uid` and `gid` the same
The special string `'inherit'` may be passed instead of a number, which will
cause this module to automatically determine the correct `uid` and/or `gid`
from the nearest existing parent directory of the target.
## `fs.withTempDir(root, fn, options) -> Promise`
### Parameters
- `root`: the directory in which to create the temporary directory
- `fn`: a function that will be called with the path to the temporary directory
- `options`
- `tmpPrefix`: a prefix to be used in the generated directory name
### Usage
The `withTempDir` function creates a temporary directory, runs the provided
function (`fn`), then removes the temporary directory and resolves or rejects
based on the result of `fn`.
```js
const fs = require('@npmcli/fs')
const os = require('os')
// this function will be called with the full path to the temporary directory
// it is called with `await` behind the scenes, so can be async if desired.
const myFunction = async (tempPath) => {
return 'done!'
}
const main = async () => {
const result = await fs.withTempDir(os.tmpdir(), myFunction)
// result === 'done!'
}
main()
```
@@ -0,0 +1,20 @@
// given an input that may or may not be an object, return an object that has
// a copy of every defined property listed in 'copy'. if the input is not an
// object, assign it to the property named by 'wrap'
const getOptions = (input, { copy, wrap }) => {
const result = {}
if (input && typeof input === 'object') {
for (const prop of copy) {
if (input[prop] !== undefined) {
result[prop] = input[prop]
}
}
} else {
result[wrap] = input
}
return result
}
module.exports = getOptions
@@ -0,0 +1,9 @@
const semver = require('semver')
const satisfies = (range) => {
return semver.satisfies(process.version, range, { includePrerelease: true })
}
module.exports = {
satisfies,
}
@@ -0,0 +1,96 @@
const { dirname, resolve } = require('path')
const url = require('url')
const fs = require('../fs.js')
// given a path, find the owner of the nearest parent
const find = (path) => {
// if we have no getuid, permissions are irrelevant on this platform
if (!process.getuid) {
return {}
}
// fs methods accept URL objects with a scheme of file: so we need to unwrap
// those into an actual path string before we can resolve it
const resolved = path != null && path.href && path.origin
? resolve(url.fileURLToPath(path))
: resolve(path)
let stat
try {
stat = fs.lstatSync(resolved)
} finally {
// if we got a stat, return its contents
if (stat) {
return { uid: stat.uid, gid: stat.gid }
}
// try the parent directory
if (resolved !== dirname(resolved)) {
return find(dirname(resolved))
}
// no more parents, never got a stat, just return an empty object
return {}
}
}
// given a path, uid, and gid update the ownership of the path if necessary
const update = (path, uid, gid) => {
// nothing to update, just exit
if (uid === undefined && gid === undefined) {
return
}
try {
// see if the permissions are already the same, if they are we don't
// need to do anything, so return early
const stat = fs.statSync(path)
if (uid === stat.uid && gid === stat.gid) {
return
}
} catch {
// ignore errors
}
try {
fs.chownSync(path, uid, gid)
} catch {
// ignore errors
}
}
// accepts a `path` and the `owner` property of an options object and normalizes
// it into an object with numerical `uid` and `gid`
const validate = (path, input) => {
let uid
let gid
if (typeof input === 'string' || typeof input === 'number') {
uid = input
gid = input
} else if (input && typeof input === 'object') {
uid = input.uid
gid = input.gid
}
if (uid === 'inherit' || gid === 'inherit') {
const owner = find(path)
if (uid === 'inherit') {
uid = owner.uid
}
if (gid === 'inherit') {
gid = owner.gid
}
}
return { uid, gid }
}
module.exports = {
find,
update,
validate,
}
@@ -0,0 +1,96 @@
const { dirname, resolve } = require('path')
const url = require('url')
const fs = require('../fs.js')
// given a path, find the owner of the nearest parent
const find = async (path) => {
// if we have no getuid, permissions are irrelevant on this platform
if (!process.getuid) {
return {}
}
// fs methods accept URL objects with a scheme of file: so we need to unwrap
// those into an actual path string before we can resolve it
const resolved = path != null && path.href && path.origin
? resolve(url.fileURLToPath(path))
: resolve(path)
let stat
try {
stat = await fs.lstat(resolved)
} finally {
// if we got a stat, return its contents
if (stat) {
return { uid: stat.uid, gid: stat.gid }
}
// try the parent directory
if (resolved !== dirname(resolved)) {
return find(dirname(resolved))
}
// no more parents, never got a stat, just return an empty object
return {}
}
}
// given a path, uid, and gid update the ownership of the path if necessary
const update = async (path, uid, gid) => {
// nothing to update, just exit
if (uid === undefined && gid === undefined) {
return
}
try {
// see if the permissions are already the same, if they are we don't
// need to do anything, so return early
const stat = await fs.stat(path)
if (uid === stat.uid && gid === stat.gid) {
return
}
} catch {
// ignore errors
}
try {
await fs.chown(path, uid, gid)
} catch {
// ignore errors
}
}
// accepts a `path` and the `owner` property of an options object and normalizes
// it into an object with numerical `uid` and `gid`
const validate = async (path, input) => {
let uid
let gid
if (typeof input === 'string' || typeof input === 'number') {
uid = input
gid = input
} else if (input && typeof input === 'object') {
uid = input.uid
gid = input.gid
}
if (uid === 'inherit' || gid === 'inherit') {
const owner = await find(path)
if (uid === 'inherit') {
uid = owner.uid
}
if (gid === 'inherit') {
gid = owner.gid
}
}
return { uid, gid }
}
module.exports = {
find,
update,
validate,
}
@@ -0,0 +1,16 @@
const fs = require('./fs.js')
const getOptions = require('./common/get-options.js')
const withOwner = require('./with-owner.js')
const copyFile = async (src, dest, opts) => {
const options = getOptions(opts, {
copy: ['mode'],
wrap: 'mode',
})
// the node core method as of 16.5.0 does not support the mode being in an
// object, so we have to pass the mode value directly
return withOwner(dest, () => fs.copyFile(src, dest, options.mode), opts)
}
module.exports = copyFile
@@ -0,0 +1,15 @@
(The MIT License)
Copyright (c) 2011-2017 JP Richardson
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files
(the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify,
merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS
OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,22 @@
const fs = require('../fs.js')
const getOptions = require('../common/get-options.js')
const node = require('../common/node.js')
const polyfill = require('./polyfill.js')
// node 16.7.0 added fs.cp
const useNative = node.satisfies('>=16.7.0')
const cp = async (src, dest, opts) => {
const options = getOptions(opts, {
copy: ['dereference', 'errorOnExist', 'filter', 'force', 'preserveTimestamps', 'recursive'],
})
// the polyfill is tested separately from this module, no need to hack
// process.version to try to trigger it just for coverage
// istanbul ignore next
return useNative
? fs.cp(src, dest, options)
: polyfill(src, dest, options)
}
module.exports = cp
@@ -0,0 +1,428 @@
// this file is a modified version of the code in node 17.2.0
// which is, in turn, a modified version of the fs-extra module on npm
// node core changes:
// - Use of the assert module has been replaced with core's error system.
// - All code related to the glob dependency has been removed.
// - Bring your own custom fs module is not currently supported.
// - Some basic code cleanup.
// changes here:
// - remove all callback related code
// - drop sync support
// - change assertions back to non-internal methods (see options.js)
// - throws ENOTDIR when rmdir gets an ENOENT for a path that exists in Windows
'use strict'
const {
ERR_FS_CP_DIR_TO_NON_DIR,
ERR_FS_CP_EEXIST,
ERR_FS_CP_EINVAL,
ERR_FS_CP_FIFO_PIPE,
ERR_FS_CP_NON_DIR_TO_DIR,
ERR_FS_CP_SOCKET,
ERR_FS_CP_SYMLINK_TO_SUBDIRECTORY,
ERR_FS_CP_UNKNOWN,
ERR_FS_EISDIR,
ERR_INVALID_ARG_TYPE,
} = require('../errors.js')
const {
constants: {
errno: {
EEXIST,
EISDIR,
EINVAL,
ENOTDIR,
},
},
} = require('os')
const {
chmod,
copyFile,
lstat,
mkdir,
readdir,
readlink,
stat,
symlink,
unlink,
utimes,
} = require('../fs.js')
const {
dirname,
isAbsolute,
join,
parse,
resolve,
sep,
toNamespacedPath,
} = require('path')
const { fileURLToPath } = require('url')
const defaultOptions = {
dereference: false,
errorOnExist: false,
filter: undefined,
force: true,
preserveTimestamps: false,
recursive: false,
}
async function cp (src, dest, opts) {
if (opts != null && typeof opts !== 'object') {
throw new ERR_INVALID_ARG_TYPE('options', ['Object'], opts)
}
return cpFn(
toNamespacedPath(getValidatedPath(src)),
toNamespacedPath(getValidatedPath(dest)),
{ ...defaultOptions, ...opts })
}
function getValidatedPath (fileURLOrPath) {
const path = fileURLOrPath != null && fileURLOrPath.href
&& fileURLOrPath.origin
? fileURLToPath(fileURLOrPath)
: fileURLOrPath
return path
}
async function cpFn (src, dest, opts) {
// Warn about using preserveTimestamps on 32-bit node
// istanbul ignore next
if (opts.preserveTimestamps && process.arch === 'ia32') {
const warning = 'Using the preserveTimestamps option in 32-bit ' +
'node is not recommended'
process.emitWarning(warning, 'TimestampPrecisionWarning')
}
const stats = await checkPaths(src, dest, opts)
const { srcStat, destStat } = stats
await checkParentPaths(src, srcStat, dest)
if (opts.filter) {
return handleFilter(checkParentDir, destStat, src, dest, opts)
}
return checkParentDir(destStat, src, dest, opts)
}
async function checkPaths (src, dest, opts) {
const { 0: srcStat, 1: destStat } = await getStats(src, dest, opts)
if (destStat) {
if (areIdentical(srcStat, destStat)) {
throw new ERR_FS_CP_EINVAL({
message: 'src and dest cannot be the same',
path: dest,
syscall: 'cp',
errno: EINVAL,
})
}
if (srcStat.isDirectory() && !destStat.isDirectory()) {
throw new ERR_FS_CP_DIR_TO_NON_DIR({
message: `cannot overwrite directory ${src} ` +
`with non-directory ${dest}`,
path: dest,
syscall: 'cp',
errno: EISDIR,
})
}
if (!srcStat.isDirectory() && destStat.isDirectory()) {
throw new ERR_FS_CP_NON_DIR_TO_DIR({
message: `cannot overwrite non-directory ${src} ` +
`with directory ${dest}`,
path: dest,
syscall: 'cp',
errno: ENOTDIR,
})
}
}
if (srcStat.isDirectory() && isSrcSubdir(src, dest)) {
throw new ERR_FS_CP_EINVAL({
message: `cannot copy ${src} to a subdirectory of self ${dest}`,
path: dest,
syscall: 'cp',
errno: EINVAL,
})
}
return { srcStat, destStat }
}
function areIdentical (srcStat, destStat) {
return destStat.ino && destStat.dev && destStat.ino === srcStat.ino &&
destStat.dev === srcStat.dev
}
function getStats (src, dest, opts) {
const statFunc = opts.dereference ?
(file) => stat(file, { bigint: true }) :
(file) => lstat(file, { bigint: true })
return Promise.all([
statFunc(src),
statFunc(dest).catch((err) => {
// istanbul ignore next: unsure how to cover.
if (err.code === 'ENOENT') {
return null
}
// istanbul ignore next: unsure how to cover.
throw err
}),
])
}
async function checkParentDir (destStat, src, dest, opts) {
const destParent = dirname(dest)
const dirExists = await pathExists(destParent)
if (dirExists) {
return getStatsForCopy(destStat, src, dest, opts)
}
await mkdir(destParent, { recursive: true })
return getStatsForCopy(destStat, src, dest, opts)
}
function pathExists (dest) {
return stat(dest).then(
() => true,
// istanbul ignore next: not sure when this would occur
(err) => (err.code === 'ENOENT' ? false : Promise.reject(err)))
}
// Recursively check if dest parent is a subdirectory of src.
// It works for all file types including symlinks since it
// checks the src and dest inodes. It starts from the deepest
// parent and stops once it reaches the src parent or the root path.
async function checkParentPaths (src, srcStat, dest) {
const srcParent = resolve(dirname(src))
const destParent = resolve(dirname(dest))
if (destParent === srcParent || destParent === parse(destParent).root) {
return
}
let destStat
try {
destStat = await stat(destParent, { bigint: true })
} catch (err) {
// istanbul ignore else: not sure when this would occur
if (err.code === 'ENOENT') {
return
}
// istanbul ignore next: not sure when this would occur
throw err
}
if (areIdentical(srcStat, destStat)) {
throw new ERR_FS_CP_EINVAL({
message: `cannot copy ${src} to a subdirectory of self ${dest}`,
path: dest,
syscall: 'cp',
errno: EINVAL,
})
}
return checkParentPaths(src, srcStat, destParent)
}
const normalizePathToArray = (path) =>
resolve(path).split(sep).filter(Boolean)
// Return true if dest is a subdir of src, otherwise false.
// It only checks the path strings.
function isSrcSubdir (src, dest) {
const srcArr = normalizePathToArray(src)
const destArr = normalizePathToArray(dest)
return srcArr.every((cur, i) => destArr[i] === cur)
}
async function handleFilter (onInclude, destStat, src, dest, opts, cb) {
const include = await opts.filter(src, dest)
if (include) {
return onInclude(destStat, src, dest, opts, cb)
}
}
function startCopy (destStat, src, dest, opts) {
if (opts.filter) {
return handleFilter(getStatsForCopy, destStat, src, dest, opts)
}
return getStatsForCopy(destStat, src, dest, opts)
}
async function getStatsForCopy (destStat, src, dest, opts) {
const statFn = opts.dereference ? stat : lstat
const srcStat = await statFn(src)
// istanbul ignore else: can't portably test FIFO
if (srcStat.isDirectory() && opts.recursive) {
return onDir(srcStat, destStat, src, dest, opts)
} else if (srcStat.isDirectory()) {
throw new ERR_FS_EISDIR({
message: `${src} is a directory (not copied)`,
path: src,
syscall: 'cp',
errno: EINVAL,
})
} else if (srcStat.isFile() ||
srcStat.isCharacterDevice() ||
srcStat.isBlockDevice()) {
return onFile(srcStat, destStat, src, dest, opts)
} else if (srcStat.isSymbolicLink()) {
return onLink(destStat, src, dest)
} else if (srcStat.isSocket()) {
throw new ERR_FS_CP_SOCKET({
message: `cannot copy a socket file: ${dest}`,
path: dest,
syscall: 'cp',
errno: EINVAL,
})
} else if (srcStat.isFIFO()) {
throw new ERR_FS_CP_FIFO_PIPE({
message: `cannot copy a FIFO pipe: ${dest}`,
path: dest,
syscall: 'cp',
errno: EINVAL,
})
}
// istanbul ignore next: should be unreachable
throw new ERR_FS_CP_UNKNOWN({
message: `cannot copy an unknown file type: ${dest}`,
path: dest,
syscall: 'cp',
errno: EINVAL,
})
}
function onFile (srcStat, destStat, src, dest, opts) {
if (!destStat) {
return _copyFile(srcStat, src, dest, opts)
}
return mayCopyFile(srcStat, src, dest, opts)
}
async function mayCopyFile (srcStat, src, dest, opts) {
if (opts.force) {
await unlink(dest)
return _copyFile(srcStat, src, dest, opts)
} else if (opts.errorOnExist) {
throw new ERR_FS_CP_EEXIST({
message: `${dest} already exists`,
path: dest,
syscall: 'cp',
errno: EEXIST,
})
}
}
async function _copyFile (srcStat, src, dest, opts) {
await copyFile(src, dest)
if (opts.preserveTimestamps) {
return handleTimestampsAndMode(srcStat.mode, src, dest)
}
return setDestMode(dest, srcStat.mode)
}
async function handleTimestampsAndMode (srcMode, src, dest) {
// Make sure the file is writable before setting the timestamp
// otherwise open fails with EPERM when invoked with 'r+'
// (through utimes call)
if (fileIsNotWritable(srcMode)) {
await makeFileWritable(dest, srcMode)
return setDestTimestampsAndMode(srcMode, src, dest)
}
return setDestTimestampsAndMode(srcMode, src, dest)
}
function fileIsNotWritable (srcMode) {
return (srcMode & 0o200) === 0
}
function makeFileWritable (dest, srcMode) {
return setDestMode(dest, srcMode | 0o200)
}
async function setDestTimestampsAndMode (srcMode, src, dest) {
await setDestTimestamps(src, dest)
return setDestMode(dest, srcMode)
}
function setDestMode (dest, srcMode) {
return chmod(dest, srcMode)
}
async function setDestTimestamps (src, dest) {
// The initial srcStat.atime cannot be trusted
// because it is modified by the read(2) system call
// (See https://nodejs.org/api/fs.html#fs_stat_time_values)
const updatedSrcStat = await stat(src)
return utimes(dest, updatedSrcStat.atime, updatedSrcStat.mtime)
}
function onDir (srcStat, destStat, src, dest, opts) {
if (!destStat) {
return mkDirAndCopy(srcStat.mode, src, dest, opts)
}
return copyDir(src, dest, opts)
}
async function mkDirAndCopy (srcMode, src, dest, opts) {
await mkdir(dest)
await copyDir(src, dest, opts)
return setDestMode(dest, srcMode)
}
async function copyDir (src, dest, opts) {
const dir = await readdir(src)
for (let i = 0; i < dir.length; i++) {
const item = dir[i]
const srcItem = join(src, item)
const destItem = join(dest, item)
const { destStat } = await checkPaths(srcItem, destItem, opts)
await startCopy(destStat, srcItem, destItem, opts)
}
}
async function onLink (destStat, src, dest) {
let resolvedSrc = await readlink(src)
if (!isAbsolute(resolvedSrc)) {
resolvedSrc = resolve(dirname(src), resolvedSrc)
}
if (!destStat) {
return symlink(resolvedSrc, dest)
}
let resolvedDest
try {
resolvedDest = await readlink(dest)
} catch (err) {
// Dest exists and is a regular file or directory,
// Windows may throw UNKNOWN error. If dest already exists,
// fs throws error anyway, so no need to guard against it here.
// istanbul ignore next: can only test on windows
if (err.code === 'EINVAL' || err.code === 'UNKNOWN') {
return symlink(resolvedSrc, dest)
}
// istanbul ignore next: should not be possible
throw err
}
if (!isAbsolute(resolvedDest)) {
resolvedDest = resolve(dirname(dest), resolvedDest)
}
if (isSrcSubdir(resolvedSrc, resolvedDest)) {
throw new ERR_FS_CP_EINVAL({
message: `cannot copy ${resolvedSrc} to a subdirectory of self ` +
`${resolvedDest}`,
path: dest,
syscall: 'cp',
errno: EINVAL,
})
}
// Do not copy if src is a subdir of dest since unlinking
// dest in this case would result in removing src contents
// and therefore a broken symlink would be created.
const srcStat = await stat(src)
if (srcStat.isDirectory() && isSrcSubdir(resolvedDest, resolvedSrc)) {
throw new ERR_FS_CP_SYMLINK_TO_SUBDIRECTORY({
message: `cannot overwrite ${resolvedDest} with ${resolvedSrc}`,
path: dest,
syscall: 'cp',
errno: EINVAL,
})
}
return copyLink(resolvedSrc, dest)
}
async function copyLink (resolvedSrc, dest) {
await unlink(dest)
return symlink(resolvedSrc, dest)
}
module.exports = cp
@@ -0,0 +1,129 @@
'use strict'
const { inspect } = require('util')
// adapted from node's internal/errors
// https://github.com/nodejs/node/blob/c8a04049/lib/internal/errors.js
// close copy of node's internal SystemError class.
class SystemError {
constructor (code, prefix, context) {
// XXX context.code is undefined in all constructors used in cp/polyfill
// that may be a bug copied from node, maybe the constructor should use
// `code` not `errno`? nodejs/node#41104
let message = `${prefix}: ${context.syscall} returned ` +
`${context.code} (${context.message})`
if (context.path !== undefined) {
message += ` ${context.path}`
}
if (context.dest !== undefined) {
message += ` => ${context.dest}`
}
this.code = code
Object.defineProperties(this, {
name: {
value: 'SystemError',
enumerable: false,
writable: true,
configurable: true,
},
message: {
value: message,
enumerable: false,
writable: true,
configurable: true,
},
info: {
value: context,
enumerable: true,
configurable: true,
writable: false,
},
errno: {
get () {
return context.errno
},
set (value) {
context.errno = value
},
enumerable: true,
configurable: true,
},
syscall: {
get () {
return context.syscall
},
set (value) {
context.syscall = value
},
enumerable: true,
configurable: true,
},
})
if (context.path !== undefined) {
Object.defineProperty(this, 'path', {
get () {
return context.path
},
set (value) {
context.path = value
},
enumerable: true,
configurable: true,
})
}
if (context.dest !== undefined) {
Object.defineProperty(this, 'dest', {
get () {
return context.dest
},
set (value) {
context.dest = value
},
enumerable: true,
configurable: true,
})
}
}
toString () {
return `${this.name} [${this.code}]: ${this.message}`
}
[Symbol.for('nodejs.util.inspect.custom')] (_recurseTimes, ctx) {
return inspect(this, {
...ctx,
getters: true,
customInspect: false,
})
}
}
function E (code, message) {
module.exports[code] = class NodeError extends SystemError {
constructor (ctx) {
super(code, message, ctx)
}
}
}
E('ERR_FS_CP_DIR_TO_NON_DIR', 'Cannot overwrite directory with non-directory')
E('ERR_FS_CP_EEXIST', 'Target already exists')
E('ERR_FS_CP_EINVAL', 'Invalid src or dest')
E('ERR_FS_CP_FIFO_PIPE', 'Cannot copy a FIFO pipe')
E('ERR_FS_CP_NON_DIR_TO_DIR', 'Cannot overwrite non-directory with directory')
E('ERR_FS_CP_SOCKET', 'Cannot copy a socket file')
E('ERR_FS_CP_SYMLINK_TO_SUBDIRECTORY', 'Cannot overwrite symlink in subdirectory of self')
E('ERR_FS_CP_UNKNOWN', 'Cannot copy an unknown file type')
E('ERR_FS_EISDIR', 'Path is a directory')
module.exports.ERR_INVALID_ARG_TYPE = class ERR_INVALID_ARG_TYPE extends Error {
constructor (name, expected, actual) {
super()
this.code = 'ERR_INVALID_ARG_TYPE'
this.message = `The ${name} argument must be ${expected}. Received ${typeof actual}`
}
}
@@ -0,0 +1,14 @@
const fs = require('fs')
const promisify = require('@gar/promisify')
const isLower = (s) => s === s.toLowerCase() && s !== s.toUpperCase()
const fsSync = Object.fromEntries(Object.entries(fs).filter(([k, v]) =>
typeof v === 'function' && (k.endsWith('Sync') || !isLower(k[0]))
))
// this module returns the core fs async fns wrapped in a proxy that promisifies
// method calls within the getter. we keep it in a separate module so that the
// overridden methods have a consistent way to get to promisified fs methods
// without creating a circular dependency. the ctors and sync methods are kept untouched
module.exports = { ...promisify(fs), ...fsSync }
@@ -0,0 +1,12 @@
module.exports = {
...require('./fs.js'),
copyFile: require('./copy-file.js'),
cp: require('./cp/index.js'),
mkdir: require('./mkdir.js'),
mkdtemp: require('./mkdtemp.js'),
rm: require('./rm/index.js'),
withTempDir: require('./with-temp-dir.js'),
withOwner: require('./with-owner.js'),
withOwnerSync: require('./with-owner-sync.js'),
writeFile: require('./write-file.js'),
}
@@ -0,0 +1,19 @@
const fs = require('./fs.js')
const getOptions = require('./common/get-options.js')
const withOwner = require('./with-owner.js')
// extends mkdir with the ability to specify an owner of the new dir
const mkdir = async (path, opts) => {
const options = getOptions(opts, {
copy: ['mode', 'recursive'],
wrap: 'mode',
})
return withOwner(
path,
() => fs.mkdir(path, options),
opts
)
}
module.exports = mkdir
@@ -0,0 +1,23 @@
const { dirname, sep } = require('path')
const fs = require('./fs.js')
const getOptions = require('./common/get-options.js')
const withOwner = require('./with-owner.js')
const mkdtemp = async (prefix, opts) => {
const options = getOptions(opts, {
copy: ['encoding'],
wrap: 'encoding',
})
// mkdtemp relies on the trailing path separator to indicate if it should
// create a directory inside of the prefix. if that's the case then the root
// we infer ownership from is the prefix itself, otherwise it's the dirname
// /tmp -> /tmpABCDEF, infers from /
// /tmp/ -> /tmp/ABCDEF, infers from /tmp
const root = prefix.endsWith(sep) ? prefix : dirname(prefix)
return withOwner(root, () => fs.mkdtemp(prefix, options), opts)
}
module.exports = mkdtemp
@@ -0,0 +1,22 @@
const fs = require('../fs.js')
const getOptions = require('../common/get-options.js')
const node = require('../common/node.js')
const polyfill = require('./polyfill.js')
// node 14.14.0 added fs.rm, which allows both the force and recursive options
const useNative = node.satisfies('>=14.14.0')
const rm = async (path, opts) => {
const options = getOptions(opts, {
copy: ['retryDelay', 'maxRetries', 'recursive', 'force'],
})
// the polyfill is tested separately from this module, no need to hack
// process.version to try to trigger it just for coverage
// istanbul ignore next
return useNative
? fs.rm(path, options)
: polyfill(path, options)
}
module.exports = rm
@@ -0,0 +1,239 @@
// this file is a modified version of the code in node core >=14.14.0
// which is, in turn, a modified version of the rimraf module on npm
// node core changes:
// - Use of the assert module has been replaced with core's error system.
// - All code related to the glob dependency has been removed.
// - Bring your own custom fs module is not currently supported.
// - Some basic code cleanup.
// changes here:
// - remove all callback related code
// - drop sync support
// - change assertions back to non-internal methods (see options.js)
// - throws ENOTDIR when rmdir gets an ENOENT for a path that exists in Windows
const errnos = require('os').constants.errno
const { join } = require('path')
const fs = require('../fs.js')
// error codes that mean we need to remove contents
const notEmptyCodes = new Set([
'ENOTEMPTY',
'EEXIST',
'EPERM',
])
// error codes we can retry later
const retryCodes = new Set([
'EBUSY',
'EMFILE',
'ENFILE',
'ENOTEMPTY',
'EPERM',
])
const isWindows = process.platform === 'win32'
const defaultOptions = {
retryDelay: 100,
maxRetries: 0,
recursive: false,
force: false,
}
// this is drastically simplified, but should be roughly equivalent to what
// node core throws
class ERR_FS_EISDIR extends Error {
constructor (path) {
super()
this.info = {
code: 'EISDIR',
message: 'is a directory',
path,
syscall: 'rm',
errno: errnos.EISDIR,
}
this.name = 'SystemError'
this.code = 'ERR_FS_EISDIR'
this.errno = errnos.EISDIR
this.syscall = 'rm'
this.path = path
this.message = `Path is a directory: ${this.syscall} returned ` +
`${this.info.code} (is a directory) ${path}`
}
toString () {
return `${this.name} [${this.code}]: ${this.message}`
}
}
class ENOTDIR extends Error {
constructor (path) {
super()
this.name = 'Error'
this.code = 'ENOTDIR'
this.errno = errnos.ENOTDIR
this.syscall = 'rmdir'
this.path = path
this.message = `not a directory, ${this.syscall} '${this.path}'`
}
toString () {
return `${this.name}: ${this.code}: ${this.message}`
}
}
// force is passed separately here because we respect it for the first entry
// into rimraf only, any further calls that are spawned as a result (i.e. to
// delete content within the target) will ignore ENOENT errors
const rimraf = async (path, options, isTop = false) => {
const force = isTop ? options.force : true
const stat = await fs.lstat(path)
.catch((err) => {
// we only ignore ENOENT if we're forcing this call
if (err.code === 'ENOENT' && force) {
return
}
if (isWindows && err.code === 'EPERM') {
return fixEPERM(path, options, err, isTop)
}
throw err
})
// no stat object here means either lstat threw an ENOENT, or lstat threw
// an EPERM and the fixPERM function took care of things. either way, we're
// already done, so return early
if (!stat) {
return
}
if (stat.isDirectory()) {
return rmdir(path, options, null, isTop)
}
return fs.unlink(path)
.catch((err) => {
if (err.code === 'ENOENT' && force) {
return
}
if (err.code === 'EISDIR') {
return rmdir(path, options, err, isTop)
}
if (err.code === 'EPERM') {
// in windows, we handle this through fixEPERM which will also try to
// delete things again. everywhere else since deleting the target as a
// file didn't work we go ahead and try to delete it as a directory
return isWindows
? fixEPERM(path, options, err, isTop)
: rmdir(path, options, err, isTop)
}
throw err
})
}
const fixEPERM = async (path, options, originalErr, isTop) => {
const force = isTop ? options.force : true
const targetMissing = await fs.chmod(path, 0o666)
.catch((err) => {
if (err.code === 'ENOENT' && force) {
return true
}
throw originalErr
})
// got an ENOENT above, return now. no file = no problem
if (targetMissing) {
return
}
// this function does its own lstat rather than calling rimraf again to avoid
// infinite recursion for a repeating EPERM
const stat = await fs.lstat(path)
.catch((err) => {
if (err.code === 'ENOENT' && force) {
return
}
throw originalErr
})
if (!stat) {
return
}
if (stat.isDirectory()) {
return rmdir(path, options, originalErr, isTop)
}
return fs.unlink(path)
}
const rmdir = async (path, options, originalErr, isTop) => {
if (!options.recursive && isTop) {
throw originalErr || new ERR_FS_EISDIR(path)
}
const force = isTop ? options.force : true
return fs.rmdir(path)
.catch(async (err) => {
// in Windows, calling rmdir on a file path will fail with ENOENT rather
// than ENOTDIR. to determine if that's what happened, we have to do
// another lstat on the path. if the path isn't actually gone, we throw
// away the ENOENT and replace it with our own ENOTDIR
if (isWindows && err.code === 'ENOENT') {
const stillExists = await fs.lstat(path).then(() => true, () => false)
if (stillExists) {
err = new ENOTDIR(path)
}
}
// not there, not a problem
if (err.code === 'ENOENT' && force) {
return
}
// we may not have originalErr if lstat tells us our target is a
// directory but that changes before we actually remove it, so
// only throw it here if it's set
if (originalErr && err.code === 'ENOTDIR') {
throw originalErr
}
// the directory isn't empty, remove the contents and try again
if (notEmptyCodes.has(err.code)) {
const files = await fs.readdir(path)
await Promise.all(files.map((file) => {
const target = join(path, file)
return rimraf(target, options)
}))
return fs.rmdir(path)
}
throw err
})
}
const rm = async (path, opts) => {
const options = { ...defaultOptions, ...opts }
let retries = 0
const errHandler = async (err) => {
if (retryCodes.has(err.code) && ++retries < options.maxRetries) {
const delay = retries * options.retryDelay
await promiseTimeout(delay)
return rimraf(path, options, true).catch(errHandler)
}
throw err
}
return rimraf(path, options, true).catch(errHandler)
}
const promiseTimeout = (ms) => new Promise((r) => setTimeout(r, ms))
module.exports = rm
@@ -0,0 +1,21 @@
const getOptions = require('./common/get-options.js')
const owner = require('./common/owner-sync.js')
const withOwnerSync = (path, fn, opts) => {
const options = getOptions(opts, {
copy: ['owner'],
})
const { uid, gid } = owner.validate(path, options.owner)
const result = fn({ uid, gid })
owner.update(path, uid, gid)
if (typeof result === 'string') {
owner.update(result, uid, gid)
}
return result
}
module.exports = withOwnerSync
@@ -0,0 +1,21 @@
const getOptions = require('./common/get-options.js')
const owner = require('./common/owner.js')
const withOwner = async (path, fn, opts) => {
const options = getOptions(opts, {
copy: ['owner'],
})
const { uid, gid } = await owner.validate(path, options.owner)
const result = await fn({ uid, gid })
await Promise.all([
owner.update(path, uid, gid),
typeof result === 'string' ? owner.update(result, uid, gid) : null,
])
return result
}
module.exports = withOwner
@@ -0,0 +1,41 @@
const { join, sep } = require('path')
const getOptions = require('./common/get-options.js')
const mkdir = require('./mkdir.js')
const mkdtemp = require('./mkdtemp.js')
const rm = require('./rm/index.js')
// create a temp directory, ensure its permissions match its parent, then call
// the supplied function passing it the path to the directory. clean up after
// the function finishes, whether it throws or not
const withTempDir = async (root, fn, opts) => {
const options = getOptions(opts, {
copy: ['tmpPrefix'],
})
// create the directory, and fix its ownership
await mkdir(root, { recursive: true, owner: 'inherit' })
const target = await mkdtemp(join(`${root}${sep}`, options.tmpPrefix || ''), { owner: 'inherit' })
let err
let result
try {
result = await fn(target)
} catch (_err) {
err = _err
}
try {
await rm(target, { force: true, recursive: true })
} catch {
// ignore errors
}
if (err) {
throw err
}
return result
}
module.exports = withTempDir
@@ -0,0 +1,14 @@
const fs = require('./fs.js')
const getOptions = require('./common/get-options.js')
const withOwner = require('./with-owner.js')
const writeFile = async (file, data, opts) => {
const options = getOptions(opts, {
copy: ['encoding', 'mode', 'flag', 'signal'],
wrap: 'encoding',
})
return withOwner(file, () => fs.writeFile(file, data, options), opts)
}
module.exports = writeFile
@@ -0,0 +1,50 @@
{
"name": "@npmcli/fs",
"version": "2.1.2",
"description": "filesystem utilities for the npm cli",
"main": "lib/index.js",
"files": [
"bin/",
"lib/"
],
"scripts": {
"preversion": "npm test",
"postversion": "npm publish",
"prepublishOnly": "git push origin --follow-tags",
"snap": "tap",
"test": "tap",
"npmclilint": "npmcli-lint",
"lint": "eslint \"**/*.js\"",
"lintfix": "npm run lint -- --fix",
"posttest": "npm run lint",
"postsnap": "npm run lintfix --",
"postlint": "template-oss-check",
"template-oss-apply": "template-oss-apply --force"
},
"repository": {
"type": "git",
"url": "https://github.com/npm/fs.git"
},
"keywords": [
"npm",
"oss"
],
"author": "GitHub Inc.",
"license": "ISC",
"devDependencies": {
"@npmcli/eslint-config": "^3.0.1",
"@npmcli/template-oss": "3.5.0",
"tap": "^16.0.1"
},
"dependencies": {
"@gar/promisify": "^1.1.3",
"semver": "^7.3.5"
},
"engines": {
"node": "^12.13.0 || ^14.15.0 || >=16.0.0"
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "3.5.0"
}
}
@@ -0,0 +1,16 @@
ISC License
Copyright (c) npm, Inc.
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE COPYRIGHT HOLDER DISCLAIMS
ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE
USE OR PERFORMANCE OF THIS SOFTWARE.
+716
View File
@@ -0,0 +1,716 @@
# cacache [![npm version](https://img.shields.io/npm/v/cacache.svg)](https://npm.im/cacache) [![license](https://img.shields.io/npm/l/cacache.svg)](https://npm.im/cacache) [![Travis](https://img.shields.io/travis/npm/cacache.svg)](https://travis-ci.org/npm/cacache) [![AppVeyor](https://ci.appveyor.com/api/projects/status/github/npm/cacache?svg=true)](https://ci.appveyor.com/project/npm/cacache) [![Coverage Status](https://coveralls.io/repos/github/npm/cacache/badge.svg?branch=latest)](https://coveralls.io/github/npm/cacache?branch=latest)
[`cacache`](https://github.com/npm/cacache) is a Node.js library for managing
local key and content address caches. It's really fast, really good at
concurrency, and it will never give you corrupted data, even if cache files
get corrupted or manipulated.
On systems that support user and group settings on files, cacache will
match the `uid` and `gid` values to the folder where the cache lives, even
when running as `root`.
It was written to be used as [npm](https://npm.im)'s local cache, but can
just as easily be used on its own.
## Install
`$ npm install --save cacache`
## Table of Contents
* [Example](#example)
* [Features](#features)
* [Contributing](#contributing)
* [API](#api)
* [Using localized APIs](#localized-api)
* Reading
* [`ls`](#ls)
* [`ls.stream`](#ls-stream)
* [`get`](#get-data)
* [`get.stream`](#get-stream)
* [`get.info`](#get-info)
* [`get.hasContent`](#get-hasContent)
* Writing
* [`put`](#put-data)
* [`put.stream`](#put-stream)
* [`rm.all`](#rm-all)
* [`rm.entry`](#rm-entry)
* [`rm.content`](#rm-content)
* [`index.compact`](#index-compact)
* [`index.insert`](#index-insert)
* Utilities
* [`clearMemoized`](#clear-memoized)
* [`tmp.mkdir`](#tmp-mkdir)
* [`tmp.withTmp`](#with-tmp)
* Integrity
* [Subresource Integrity](#integrity)
* [`verify`](#verify)
* [`verify.lastRun`](#verify-last-run)
### Example
```javascript
const cacache = require('cacache')
const fs = require('fs')
const tarball = '/path/to/mytar.tgz'
const cachePath = '/tmp/my-toy-cache'
const key = 'my-unique-key-1234'
// Cache it! Use `cachePath` as the root of the content cache
cacache.put(cachePath, key, '10293801983029384').then(integrity => {
console.log(`Saved content to ${cachePath}.`)
})
const destination = '/tmp/mytar.tgz'
// Copy the contents out of the cache and into their destination!
// But this time, use stream instead!
cacache.get.stream(
cachePath, key
).pipe(
fs.createWriteStream(destination)
).on('finish', () => {
console.log('done extracting!')
})
// The same thing, but skip the key index.
cacache.get.byDigest(cachePath, integrityHash).then(data => {
fs.writeFile(destination, data, err => {
console.log('tarball data fetched based on its sha512sum and written out!')
})
})
```
### Features
* Extraction by key or by content address (shasum, etc)
* [Subresource Integrity](#integrity) web standard support
* Multi-hash support - safely host sha1, sha512, etc, in a single cache
* Automatic content deduplication
* Fault tolerance (immune to corruption, partial writes, process races, etc)
* Consistency guarantees on read and write (full data verification)
* Lockless, high-concurrency cache access
* Streaming support
* Promise support
* Fast -- sub-millisecond reads and writes including verification
* Arbitrary metadata storage
* Garbage collection and additional offline verification
* Thorough test coverage
* There's probably a bloom filter in there somewhere. Those are cool, right? 🤔
### Contributing
The cacache team enthusiastically welcomes contributions and project participation! There's a bunch of things you can do if you want to contribute! The [Contributor Guide](CONTRIBUTING.md) has all the information you need for everything from reporting bugs to contributing entire new features. Please don't hesitate to jump in if you'd like to, or even ask us questions if something isn't clear.
All participants and maintainers in this project are expected to follow [Code of Conduct](CODE_OF_CONDUCT.md), and just generally be excellent to each other.
Please refer to the [Changelog](CHANGELOG.md) for project history details, too.
Happy hacking!
### API
#### <a name="ls"></a> `> cacache.ls(cache) -> Promise<Object>`
Lists info for all entries currently in the cache as a single large object. Each
entry in the object will be keyed by the unique index key, with corresponding
[`get.info`](#get-info) objects as the values.
##### Example
```javascript
cacache.ls(cachePath).then(console.log)
// Output
{
'my-thing': {
key: 'my-thing',
integrity: 'sha512-BaSe64/EnCoDED+HAsh=='
path: '.testcache/content/deadbeef', // joined with `cachePath`
time: 12345698490,
size: 4023948,
metadata: {
name: 'blah',
version: '1.2.3',
description: 'this was once a package but now it is my-thing'
}
},
'other-thing': {
key: 'other-thing',
integrity: 'sha1-ANothER+hasH=',
path: '.testcache/content/bada55',
time: 11992309289,
size: 111112
}
}
```
#### <a name="ls-stream"></a> `> cacache.ls.stream(cache) -> Readable`
Lists info for all entries currently in the cache as a single large object.
This works just like [`ls`](#ls), except [`get.info`](#get-info) entries are
returned as `'data'` events on the returned stream.
##### Example
```javascript
cacache.ls.stream(cachePath).on('data', console.log)
// Output
{
key: 'my-thing',
integrity: 'sha512-BaSe64HaSh',
path: '.testcache/content/deadbeef', // joined with `cachePath`
time: 12345698490,
size: 13423,
metadata: {
name: 'blah',
version: '1.2.3',
description: 'this was once a package but now it is my-thing'
}
}
{
key: 'other-thing',
integrity: 'whirlpool-WoWSoMuchSupport',
path: '.testcache/content/bada55',
time: 11992309289,
size: 498023984029
}
{
...
}
```
#### <a name="get-data"></a> `> cacache.get(cache, key, [opts]) -> Promise({data, metadata, integrity})`
Returns an object with the cached data, digest, and metadata identified by
`key`. The `data` property of this object will be a `Buffer` instance that
presumably holds some data that means something to you. I'm sure you know what
to do with it! cacache just won't care.
`integrity` is a [Subresource
Integrity](#integrity)
string. That is, a string that can be used to verify `data`, which looks like
`<hash-algorithm>-<base64-integrity-hash>`.
If there is no content identified by `key`, or if the locally-stored data does
not pass the validity checksum, the promise will be rejected.
A sub-function, `get.byDigest` may be used for identical behavior, except lookup
will happen by integrity hash, bypassing the index entirely. This version of the
function *only* returns `data` itself, without any wrapper.
See: [options](#get-options)
##### Note
This function loads the entire cache entry into memory before returning it. If
you're dealing with Very Large data, consider using [`get.stream`](#get-stream)
instead.
##### Example
```javascript
// Look up by key
cache.get(cachePath, 'my-thing').then(console.log)
// Output:
{
metadata: {
thingName: 'my'
},
integrity: 'sha512-BaSe64HaSh',
data: Buffer#<deadbeef>,
size: 9320
}
// Look up by digest
cache.get.byDigest(cachePath, 'sha512-BaSe64HaSh').then(console.log)
// Output:
Buffer#<deadbeef>
```
#### <a name="get-stream"></a> `> cacache.get.stream(cache, key, [opts]) -> Readable`
Returns a [Readable Stream](https://nodejs.org/api/stream.html#stream_readable_streams) of the cached data identified by `key`.
If there is no content identified by `key`, or if the locally-stored data does
not pass the validity checksum, an error will be emitted.
`metadata` and `integrity` events will be emitted before the stream closes, if
you need to collect that extra data about the cached entry.
A sub-function, `get.stream.byDigest` may be used for identical behavior,
except lookup will happen by integrity hash, bypassing the index entirely. This
version does not emit the `metadata` and `integrity` events at all.
See: [options](#get-options)
##### Example
```javascript
// Look up by key
cache.get.stream(
cachePath, 'my-thing'
).on('metadata', metadata => {
console.log('metadata:', metadata)
}).on('integrity', integrity => {
console.log('integrity:', integrity)
}).pipe(
fs.createWriteStream('./x.tgz')
)
// Outputs:
metadata: { ... }
integrity: 'sha512-SoMeDIGest+64=='
// Look up by digest
cache.get.stream.byDigest(
cachePath, 'sha512-SoMeDIGest+64=='
).pipe(
fs.createWriteStream('./x.tgz')
)
```
#### <a name="get-info"></a> `> cacache.get.info(cache, key) -> Promise`
Looks up `key` in the cache index, returning information about the entry if
one exists.
##### Fields
* `key` - Key the entry was looked up under. Matches the `key` argument.
* `integrity` - [Subresource Integrity hash](#integrity) for the content this entry refers to.
* `path` - Filesystem path where content is stored, joined with `cache` argument.
* `time` - Timestamp the entry was first added on.
* `metadata` - User-assigned metadata associated with the entry/content.
##### Example
```javascript
cacache.get.info(cachePath, 'my-thing').then(console.log)
// Output
{
key: 'my-thing',
integrity: 'sha256-MUSTVERIFY+ALL/THINGS=='
path: '.testcache/content/deadbeef',
time: 12345698490,
size: 849234,
metadata: {
name: 'blah',
version: '1.2.3',
description: 'this was once a package but now it is my-thing'
}
}
```
#### <a name="get-hasContent"></a> `> cacache.get.hasContent(cache, integrity) -> Promise`
Looks up a [Subresource Integrity hash](#integrity) in the cache. If content
exists for this `integrity`, it will return an object, with the specific single integrity hash
that was found in `sri` key, and the size of the found content as `size`. If no content exists for this integrity, it will return `false`.
##### Example
```javascript
cacache.get.hasContent(cachePath, 'sha256-MUSTVERIFY+ALL/THINGS==').then(console.log)
// Output
{
sri: {
source: 'sha256-MUSTVERIFY+ALL/THINGS==',
algorithm: 'sha256',
digest: 'MUSTVERIFY+ALL/THINGS==',
options: []
},
size: 9001
}
cacache.get.hasContent(cachePath, 'sha521-NOT+IN/CACHE==').then(console.log)
// Output
false
```
##### <a name="get-options"></a> Options
##### `opts.integrity`
If present, the pre-calculated digest for the inserted content. If this option
is provided and does not match the post-insertion digest, insertion will fail
with an `EINTEGRITY` error.
##### `opts.memoize`
Default: null
If explicitly truthy, cacache will read from memory and memoize data on bulk read. If `false`, cacache will read from disk data. Reader functions by default read from in-memory cache.
##### `opts.size`
If provided, the data stream will be verified to check that enough data was
passed through. If there's more or less data than expected, insertion will fail
with an `EBADSIZE` error.
#### <a name="put-data"></a> `> cacache.put(cache, key, data, [opts]) -> Promise`
Inserts data passed to it into the cache. The returned Promise resolves with a
digest (generated according to [`opts.algorithms`](#optsalgorithms)) after the
cache entry has been successfully written.
See: [options](#put-options)
##### Example
```javascript
fetch(
'https://registry.npmjs.org/cacache/-/cacache-1.0.0.tgz'
).then(data => {
return cacache.put(cachePath, 'registry.npmjs.org|cacache@1.0.0', data)
}).then(integrity => {
console.log('integrity hash is', integrity)
})
```
#### <a name="put-stream"></a> `> cacache.put.stream(cache, key, [opts]) -> Writable`
Returns a [Writable
Stream](https://nodejs.org/api/stream.html#stream_writable_streams) that inserts
data written to it into the cache. Emits an `integrity` event with the digest of
written contents when it succeeds.
See: [options](#put-options)
##### Example
```javascript
request.get(
'https://registry.npmjs.org/cacache/-/cacache-1.0.0.tgz'
).pipe(
cacache.put.stream(
cachePath, 'registry.npmjs.org|cacache@1.0.0'
).on('integrity', d => console.log(`integrity digest is ${d}`))
)
```
##### <a name="put-options"></a> Options
##### `opts.metadata`
Arbitrary metadata to be attached to the inserted key.
##### `opts.size`
If provided, the data stream will be verified to check that enough data was
passed through. If there's more or less data than expected, insertion will fail
with an `EBADSIZE` error.
##### `opts.integrity`
If present, the pre-calculated digest for the inserted content. If this option
is provided and does not match the post-insertion digest, insertion will fail
with an `EINTEGRITY` error.
`algorithms` has no effect if this option is present.
##### `opts.integrityEmitter`
*Streaming only* If present, uses the provided event emitter as a source of
truth for both integrity and size. This allows use cases where integrity is
already being calculated outside of cacache to reuse that data instead of
calculating it a second time.
The emitter must emit both the `'integrity'` and `'size'` events.
NOTE: If this option is provided, you must verify that you receive the correct
integrity value yourself and emit an `'error'` event if there is a mismatch.
[ssri Integrity Streams](https://github.com/npm/ssri#integrity-stream) do this for you when given an expected integrity.
##### `opts.algorithms`
Default: ['sha512']
Hashing algorithms to use when calculating the [subresource integrity
digest](#integrity)
for inserted data. Can use any algorithm listed in `crypto.getHashes()` or
`'omakase'`/`'お任せします'` to pick a random hash algorithm on each insertion. You
may also use any anagram of `'modnar'` to use this feature.
Currently only supports one algorithm at a time (i.e., an array length of
exactly `1`). Has no effect if `opts.integrity` is present.
##### `opts.memoize`
Default: null
If provided, cacache will memoize the given cache insertion in memory, bypassing
any filesystem checks for that key or digest in future cache fetches. Nothing
will be written to the in-memory cache unless this option is explicitly truthy.
If `opts.memoize` is an object or a `Map`-like (that is, an object with `get`
and `set` methods), it will be written to instead of the global memoization
cache.
Reading from disk data can be forced by explicitly passing `memoize: false` to
the reader functions, but their default will be to read from memory.
##### `opts.tmpPrefix`
Default: null
Prefix to append on the temporary directory name inside the cache's tmp dir.
#### <a name="rm-all"></a> `> cacache.rm.all(cache) -> Promise`
Clears the entire cache. Mainly by blowing away the cache directory itself.
##### Example
```javascript
cacache.rm.all(cachePath).then(() => {
console.log('THE APOCALYPSE IS UPON US 😱')
})
```
#### <a name="rm-entry"></a> `> cacache.rm.entry(cache, key, [opts]) -> Promise`
Alias: `cacache.rm`
Removes the index entry for `key`. Content will still be accessible if
requested directly by content address ([`get.stream.byDigest`](#get-stream)).
By default, this appends a new entry to the index with an integrity of `null`.
If `opts.removeFully` is set to `true` then the index file itself will be
physically deleted rather than appending a `null`.
To remove the content itself (which might still be used by other entries), use
[`rm.content`](#rm-content). Or, to safely vacuum any unused content, use
[`verify`](#verify).
##### Example
```javascript
cacache.rm.entry(cachePath, 'my-thing').then(() => {
console.log('I did not like it anyway')
})
```
#### <a name="rm-content"></a> `> cacache.rm.content(cache, integrity) -> Promise`
Removes the content identified by `integrity`. Any index entries referring to it
will not be usable again until the content is re-added to the cache with an
identical digest.
##### Example
```javascript
cacache.rm.content(cachePath, 'sha512-SoMeDIGest/IN+BaSE64==').then(() => {
console.log('data for my-thing is gone!')
})
```
#### <a name="index-compact"></a> `> cacache.index.compact(cache, key, matchFn, [opts]) -> Promise`
Uses `matchFn`, which must be a synchronous function that accepts two entries
and returns a boolean indicating whether or not the two entries match, to
deduplicate all entries in the cache for the given `key`.
If `opts.validateEntry` is provided, it will be called as a function with the
only parameter being a single index entry. The function must return a Boolean,
if it returns `true` the entry is considered valid and will be kept in the index,
if it returns `false` the entry will be removed from the index.
If `opts.validateEntry` is not provided, however, every entry in the index will
be deduplicated and kept until the first `null` integrity is reached, removing
all entries that were written before the `null`.
The deduplicated list of entries is both written to the index, replacing the
existing content, and returned in the Promise.
#### <a name="index-insert"></a> `> cacache.index.insert(cache, key, integrity, opts) -> Promise`
Writes an index entry to the cache for the given `key` without writing content.
It is assumed if you are using this method, you have already stored the content
some other way and you only wish to add a new index to that content. The `metadata`
and `size` properties are read from `opts` and used as part of the index entry.
Returns a Promise resolving to the newly added entry.
#### <a name="clear-memoized"></a> `> cacache.clearMemoized()`
Completely resets the in-memory entry cache.
#### <a name="tmp-mkdir"></a> `> tmp.mkdir(cache, opts) -> Promise<Path>`
Returns a unique temporary directory inside the cache's `tmp` dir. This
directory will use the same safe user assignment that all the other stuff use.
Once the directory is made, it's the user's responsibility that all files
within are given the appropriate `gid`/`uid` ownership settings to match
the rest of the cache. If not, you can ask cacache to do it for you by
calling [`tmp.fix()`](#tmp-fix), which will fix all tmp directory
permissions.
If you want automatic cleanup of this directory, use
[`tmp.withTmp()`](#with-tpm)
See: [options](#tmp-options)
##### Example
```javascript
cacache.tmp.mkdir(cache).then(dir => {
fs.writeFile(path.join(dir, 'blablabla'), Buffer#<1234>, ...)
})
```
#### <a name="tmp-fix"></a> `> tmp.fix(cache) -> Promise`
Sets the `uid` and `gid` properties on all files and folders within the tmp
folder to match the rest of the cache.
Use this after manually writing files into [`tmp.mkdir`](#tmp-mkdir) or
[`tmp.withTmp`](#with-tmp).
##### Example
```javascript
cacache.tmp.mkdir(cache).then(dir => {
writeFile(path.join(dir, 'file'), someData).then(() => {
// make sure we didn't just put a root-owned file in the cache
cacache.tmp.fix().then(() => {
// all uids and gids match now
})
})
})
```
#### <a name="with-tmp"></a> `> tmp.withTmp(cache, opts, cb) -> Promise`
Creates a temporary directory with [`tmp.mkdir()`](#tmp-mkdir) and calls `cb`
with it. The created temporary directory will be removed when the return value
of `cb()` resolves, the tmp directory will be automatically deleted once that
promise completes.
The same caveats apply when it comes to managing permissions for the tmp dir's
contents.
See: [options](#tmp-options)
##### Example
```javascript
cacache.tmp.withTmp(cache, dir => {
return fs.writeFileAsync(path.join(dir, 'blablabla'), Buffer#<1234>, ...)
}).then(() => {
// `dir` no longer exists
})
```
##### <a name="tmp-options"></a> Options
##### `opts.tmpPrefix`
Default: null
Prefix to append on the temporary directory name inside the cache's tmp dir.
#### <a name="integrity"></a> Subresource Integrity Digests
For content verification and addressing, cacache uses strings following the
[Subresource
Integrity spec](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity).
That is, any time cacache expects an `integrity` argument or option, it
should be in the format `<hashAlgorithm>-<base64-hash>`.
One deviation from the current spec is that cacache will support any hash
algorithms supported by the underlying Node.js process. You can use
`crypto.getHashes()` to see which ones you can use.
##### Generating Digests Yourself
If you have an existing content shasum, they are generally formatted as a
hexadecimal string (that is, a sha1 would look like:
`5f5513f8822fdbe5145af33b64d8d970dcf95c6e`). In order to be compatible with
cacache, you'll need to convert this to an equivalent subresource integrity
string. For this example, the corresponding hash would be:
`sha1-X1UT+IIv2+UUWvM7ZNjZcNz5XG4=`.
If you want to generate an integrity string yourself for existing data, you can
use something like this:
```javascript
const crypto = require('crypto')
const hashAlgorithm = 'sha512'
const data = 'foobarbaz'
const integrity = (
hashAlgorithm +
'-' +
crypto.createHash(hashAlgorithm).update(data).digest('base64')
)
```
You can also use [`ssri`](https://npm.im/ssri) to have a richer set of functionality
around SRI strings, including generation, parsing, and translating from existing
hex-formatted strings.
#### <a name="verify"></a> `> cacache.verify(cache, opts) -> Promise`
Checks out and fixes up your cache:
* Cleans up corrupted or invalid index entries.
* Custom entry filtering options.
* Garbage collects any content entries not referenced by the index.
* Checks integrity for all content entries and removes invalid content.
* Fixes cache ownership.
* Removes the `tmp` directory in the cache and all its contents.
When it's done, it'll return an object with various stats about the verification
process, including amount of storage reclaimed, number of valid entries, number
of entries removed, etc.
##### <a name="verify-options"></a> Options
##### `opts.concurrency`
Default: 20
Number of concurrently read files in the filesystem while doing clean up.
##### `opts.filter`
Receives a formatted entry. Return false to remove it.
Note: might be called more than once on the same entry.
##### `opts.log`
Custom logger function:
```
log: { silly () {} }
log.silly('verify', 'verifying cache at', cache)
```
##### Example
```sh
echo somegarbage >> $CACHEPATH/content/deadbeef
```
```javascript
cacache.verify(cachePath).then(stats => {
// deadbeef collected, because of invalid checksum.
console.log('cache is much nicer now! stats:', stats)
})
```
#### <a name="verify-last-run"></a> `> cacache.verify.lastRun(cache) -> Promise`
Returns a `Date` representing the last time `cacache.verify` was run on `cache`.
##### Example
```javascript
cacache.verify(cachePath).then(() => {
cacache.verify.lastRun(cachePath).then(lastTime => {
console.log('cacache.verify was last called on' + lastTime)
})
})
```
@@ -0,0 +1,29 @@
'use strict'
const contentVer = require('../../package.json')['cache-version'].content
const hashToSegments = require('../util/hash-to-segments')
const path = require('path')
const ssri = require('ssri')
// Current format of content file path:
//
// sha512-BaSE64Hex= ->
// ~/.my-cache/content-v2/sha512/ba/da/55deadbeefc0ffee
//
module.exports = contentPath
function contentPath (cache, integrity) {
const sri = ssri.parse(integrity, { single: true })
// contentPath is the *strongest* algo given
return path.join(
contentDir(cache),
sri.algorithm,
...hashToSegments(sri.hexDigest())
)
}
module.exports.contentDir = contentDir
function contentDir (cache) {
return path.join(cache, `content-v${contentVer}`)
}
@@ -0,0 +1,241 @@
'use strict'
const fs = require('@npmcli/fs')
const fsm = require('fs-minipass')
const ssri = require('ssri')
const contentPath = require('./path')
const Pipeline = require('minipass-pipeline')
module.exports = read
const MAX_SINGLE_READ_SIZE = 64 * 1024 * 1024
async function read (cache, integrity, opts = {}) {
const { size } = opts
const { stat, cpath, sri } = await withContentSri(cache, integrity, async (cpath, sri) => {
// get size
const stat = await fs.stat(cpath)
return { stat, cpath, sri }
})
if (typeof size === 'number' && stat.size !== size) {
throw sizeError(size, stat.size)
}
if (stat.size > MAX_SINGLE_READ_SIZE) {
return readPipeline(cpath, stat.size, sri, new Pipeline()).concat()
}
const data = await fs.readFile(cpath, { encoding: null })
if (!ssri.checkData(data, sri)) {
throw integrityError(sri, cpath)
}
return data
}
const readPipeline = (cpath, size, sri, stream) => {
stream.push(
new fsm.ReadStream(cpath, {
size,
readSize: MAX_SINGLE_READ_SIZE,
}),
ssri.integrityStream({
integrity: sri,
size,
})
)
return stream
}
module.exports.sync = readSync
function readSync (cache, integrity, opts = {}) {
const { size } = opts
return withContentSriSync(cache, integrity, (cpath, sri) => {
const data = fs.readFileSync(cpath, { encoding: null })
if (typeof size === 'number' && size !== data.length) {
throw sizeError(size, data.length)
}
if (ssri.checkData(data, sri)) {
return data
}
throw integrityError(sri, cpath)
})
}
module.exports.stream = readStream
module.exports.readStream = readStream
function readStream (cache, integrity, opts = {}) {
const { size } = opts
const stream = new Pipeline()
// Set all this up to run on the stream and then just return the stream
Promise.resolve().then(async () => {
const { stat, cpath, sri } = await withContentSri(cache, integrity, async (cpath, sri) => {
// just stat to ensure it exists
const stat = await fs.stat(cpath)
return { stat, cpath, sri }
})
if (typeof size === 'number' && size !== stat.size) {
return stream.emit('error', sizeError(size, stat.size))
}
return readPipeline(cpath, stat.size, sri, stream)
}).catch(err => stream.emit('error', err))
return stream
}
module.exports.copy = copy
module.exports.copy.sync = copySync
function copy (cache, integrity, dest) {
return withContentSri(cache, integrity, (cpath, sri) => {
return fs.copyFile(cpath, dest)
})
}
function copySync (cache, integrity, dest) {
return withContentSriSync(cache, integrity, (cpath, sri) => {
return fs.copyFileSync(cpath, dest)
})
}
module.exports.hasContent = hasContent
async function hasContent (cache, integrity) {
if (!integrity) {
return false
}
try {
return await withContentSri(cache, integrity, async (cpath, sri) => {
const stat = await fs.stat(cpath)
return { size: stat.size, sri, stat }
})
} catch (err) {
if (err.code === 'ENOENT') {
return false
}
if (err.code === 'EPERM') {
/* istanbul ignore else */
if (process.platform !== 'win32') {
throw err
} else {
return false
}
}
}
}
module.exports.hasContent.sync = hasContentSync
function hasContentSync (cache, integrity) {
if (!integrity) {
return false
}
return withContentSriSync(cache, integrity, (cpath, sri) => {
try {
const stat = fs.statSync(cpath)
return { size: stat.size, sri, stat }
} catch (err) {
if (err.code === 'ENOENT') {
return false
}
if (err.code === 'EPERM') {
/* istanbul ignore else */
if (process.platform !== 'win32') {
throw err
} else {
return false
}
}
}
})
}
async function withContentSri (cache, integrity, fn) {
const sri = ssri.parse(integrity)
// If `integrity` has multiple entries, pick the first digest
// with available local data.
const algo = sri.pickAlgorithm()
const digests = sri[algo]
if (digests.length <= 1) {
const cpath = contentPath(cache, digests[0])
return fn(cpath, digests[0])
} else {
// Can't use race here because a generic error can happen before
// a ENOENT error, and can happen before a valid result
const results = await Promise.all(digests.map(async (meta) => {
try {
return await withContentSri(cache, meta, fn)
} catch (err) {
if (err.code === 'ENOENT') {
return Object.assign(
new Error('No matching content found for ' + sri.toString()),
{ code: 'ENOENT' }
)
}
return err
}
}))
// Return the first non error if it is found
const result = results.find((r) => !(r instanceof Error))
if (result) {
return result
}
// Throw the No matching content found error
const enoentError = results.find((r) => r.code === 'ENOENT')
if (enoentError) {
throw enoentError
}
// Throw generic error
throw results.find((r) => r instanceof Error)
}
}
function withContentSriSync (cache, integrity, fn) {
const sri = ssri.parse(integrity)
// If `integrity` has multiple entries, pick the first digest
// with available local data.
const algo = sri.pickAlgorithm()
const digests = sri[algo]
if (digests.length <= 1) {
const cpath = contentPath(cache, digests[0])
return fn(cpath, digests[0])
} else {
let lastErr = null
for (const meta of digests) {
try {
return withContentSriSync(cache, meta, fn)
} catch (err) {
lastErr = err
}
}
throw lastErr
}
}
function sizeError (expected, found) {
/* eslint-disable-next-line max-len */
const err = new Error(`Bad data size: expected inserted data to be ${expected} bytes, but got ${found} instead`)
err.expected = expected
err.found = found
err.code = 'EBADSIZE'
return err
}
function integrityError (sri, path) {
const err = new Error(`Integrity verification failed for ${sri} (${path})`)
err.code = 'EINTEGRITY'
err.sri = sri
err.path = path
return err
}
@@ -0,0 +1,20 @@
'use strict'
const util = require('util')
const contentPath = require('./path')
const { hasContent } = require('./read')
const rimraf = util.promisify(require('rimraf'))
module.exports = rm
async function rm (cache, integrity) {
const content = await hasContent(cache, integrity)
// ~pretty~ sure we can't end up with a content lacking sri, but be safe
if (content && content.sri) {
await rimraf(contentPath(cache, content.sri))
return true
} else {
return false
}
}
@@ -0,0 +1,189 @@
'use strict'
const events = require('events')
const util = require('util')
const contentPath = require('./path')
const fixOwner = require('../util/fix-owner')
const fs = require('@npmcli/fs')
const moveFile = require('../util/move-file')
const Minipass = require('minipass')
const Pipeline = require('minipass-pipeline')
const Flush = require('minipass-flush')
const path = require('path')
const rimraf = util.promisify(require('rimraf'))
const ssri = require('ssri')
const uniqueFilename = require('unique-filename')
const fsm = require('fs-minipass')
module.exports = write
async function write (cache, data, opts = {}) {
const { algorithms, size, integrity } = opts
if (algorithms && algorithms.length > 1) {
throw new Error('opts.algorithms only supports a single algorithm for now')
}
if (typeof size === 'number' && data.length !== size) {
throw sizeError(size, data.length)
}
const sri = ssri.fromData(data, algorithms ? { algorithms } : {})
if (integrity && !ssri.checkData(data, integrity, opts)) {
throw checksumError(integrity, sri)
}
const tmp = await makeTmp(cache, opts)
try {
await fs.writeFile(tmp.target, data, { flag: 'wx' })
await moveToDestination(tmp, cache, sri, opts)
return { integrity: sri, size: data.length }
} finally {
if (!tmp.moved) {
await rimraf(tmp.target)
}
}
}
module.exports.stream = writeStream
// writes proxied to the 'inputStream' that is passed to the Promise
// 'end' is deferred until content is handled.
class CacacheWriteStream extends Flush {
constructor (cache, opts) {
super()
this.opts = opts
this.cache = cache
this.inputStream = new Minipass()
this.inputStream.on('error', er => this.emit('error', er))
this.inputStream.on('drain', () => this.emit('drain'))
this.handleContentP = null
}
write (chunk, encoding, cb) {
if (!this.handleContentP) {
this.handleContentP = handleContent(
this.inputStream,
this.cache,
this.opts
)
}
return this.inputStream.write(chunk, encoding, cb)
}
flush (cb) {
this.inputStream.end(() => {
if (!this.handleContentP) {
const e = new Error('Cache input stream was empty')
e.code = 'ENODATA'
// empty streams are probably emitting end right away.
// defer this one tick by rejecting a promise on it.
return Promise.reject(e).catch(cb)
}
// eslint-disable-next-line promise/catch-or-return
this.handleContentP.then(
(res) => {
res.integrity && this.emit('integrity', res.integrity)
// eslint-disable-next-line promise/always-return
res.size !== null && this.emit('size', res.size)
cb()
},
(er) => cb(er)
)
})
}
}
function writeStream (cache, opts = {}) {
return new CacacheWriteStream(cache, opts)
}
async function handleContent (inputStream, cache, opts) {
const tmp = await makeTmp(cache, opts)
try {
const res = await pipeToTmp(inputStream, cache, tmp.target, opts)
await moveToDestination(
tmp,
cache,
res.integrity,
opts
)
return res
} finally {
if (!tmp.moved) {
await rimraf(tmp.target)
}
}
}
async function pipeToTmp (inputStream, cache, tmpTarget, opts) {
const outStream = new fsm.WriteStream(tmpTarget, {
flags: 'wx',
})
if (opts.integrityEmitter) {
// we need to create these all simultaneously since they can fire in any order
const [integrity, size] = await Promise.all([
events.once(opts.integrityEmitter, 'integrity').then(res => res[0]),
events.once(opts.integrityEmitter, 'size').then(res => res[0]),
new Pipeline(inputStream, outStream).promise(),
])
return { integrity, size }
}
let integrity
let size
const hashStream = ssri.integrityStream({
integrity: opts.integrity,
algorithms: opts.algorithms,
size: opts.size,
})
hashStream.on('integrity', i => {
integrity = i
})
hashStream.on('size', s => {
size = s
})
const pipeline = new Pipeline(inputStream, hashStream, outStream)
await pipeline.promise()
return { integrity, size }
}
async function makeTmp (cache, opts) {
const tmpTarget = uniqueFilename(path.join(cache, 'tmp'), opts.tmpPrefix)
await fixOwner.mkdirfix(cache, path.dirname(tmpTarget))
return {
target: tmpTarget,
moved: false,
}
}
async function moveToDestination (tmp, cache, sri, opts) {
const destination = contentPath(cache, sri)
const destDir = path.dirname(destination)
await fixOwner.mkdirfix(cache, destDir)
await moveFile(tmp.target, destination)
tmp.moved = true
await fixOwner.chownr(cache, destination)
}
function sizeError (expected, found) {
/* eslint-disable-next-line max-len */
const err = new Error(`Bad data size: expected inserted data to be ${expected} bytes, but got ${found} instead`)
err.expected = expected
err.found = found
err.code = 'EBADSIZE'
return err
}
function checksumError (expected, found) {
const err = new Error(`Integrity check failed:
Wanted: ${expected}
Found: ${found}`)
err.code = 'EINTEGRITY'
err.expected = expected
err.found = found
return err
}
@@ -0,0 +1,404 @@
'use strict'
const util = require('util')
const crypto = require('crypto')
const fs = require('@npmcli/fs')
const Minipass = require('minipass')
const path = require('path')
const ssri = require('ssri')
const uniqueFilename = require('unique-filename')
const contentPath = require('./content/path')
const fixOwner = require('./util/fix-owner')
const hashToSegments = require('./util/hash-to-segments')
const indexV = require('../package.json')['cache-version'].index
const moveFile = require('@npmcli/move-file')
const _rimraf = require('rimraf')
const rimraf = util.promisify(_rimraf)
rimraf.sync = _rimraf.sync
module.exports.NotFoundError = class NotFoundError extends Error {
constructor (cache, key) {
super(`No cache entry for ${key} found in ${cache}`)
this.code = 'ENOENT'
this.cache = cache
this.key = key
}
}
module.exports.compact = compact
async function compact (cache, key, matchFn, opts = {}) {
const bucket = bucketPath(cache, key)
const entries = await bucketEntries(bucket)
const newEntries = []
// we loop backwards because the bottom-most result is the newest
// since we add new entries with appendFile
for (let i = entries.length - 1; i >= 0; --i) {
const entry = entries[i]
// a null integrity could mean either a delete was appended
// or the user has simply stored an index that does not map
// to any content. we determine if the user wants to keep the
// null integrity based on the validateEntry function passed in options.
// if the integrity is null and no validateEntry is provided, we break
// as we consider the null integrity to be a deletion of everything
// that came before it.
if (entry.integrity === null && !opts.validateEntry) {
break
}
// if this entry is valid, and it is either the first entry or
// the newEntries array doesn't already include an entry that
// matches this one based on the provided matchFn, then we add
// it to the beginning of our list
if ((!opts.validateEntry || opts.validateEntry(entry) === true) &&
(newEntries.length === 0 ||
!newEntries.find((oldEntry) => matchFn(oldEntry, entry)))) {
newEntries.unshift(entry)
}
}
const newIndex = '\n' + newEntries.map((entry) => {
const stringified = JSON.stringify(entry)
const hash = hashEntry(stringified)
return `${hash}\t${stringified}`
}).join('\n')
const setup = async () => {
const target = uniqueFilename(path.join(cache, 'tmp'), opts.tmpPrefix)
await fixOwner.mkdirfix(cache, path.dirname(target))
return {
target,
moved: false,
}
}
const teardown = async (tmp) => {
if (!tmp.moved) {
return rimraf(tmp.target)
}
}
const write = async (tmp) => {
await fs.writeFile(tmp.target, newIndex, { flag: 'wx' })
await fixOwner.mkdirfix(cache, path.dirname(bucket))
// we use @npmcli/move-file directly here because we
// want to overwrite the existing file
await moveFile(tmp.target, bucket)
tmp.moved = true
try {
await fixOwner.chownr(cache, bucket)
} catch (err) {
if (err.code !== 'ENOENT') {
throw err
}
}
}
// write the file atomically
const tmp = await setup()
try {
await write(tmp)
} finally {
await teardown(tmp)
}
// we reverse the list we generated such that the newest
// entries come first in order to make looping through them easier
// the true passed to formatEntry tells it to keep null
// integrity values, if they made it this far it's because
// validateEntry returned true, and as such we should return it
return newEntries.reverse().map((entry) => formatEntry(cache, entry, true))
}
module.exports.insert = insert
async function insert (cache, key, integrity, opts = {}) {
const { metadata, size } = opts
const bucket = bucketPath(cache, key)
const entry = {
key,
integrity: integrity && ssri.stringify(integrity),
time: Date.now(),
size,
metadata,
}
try {
await fixOwner.mkdirfix(cache, path.dirname(bucket))
const stringified = JSON.stringify(entry)
// NOTE - Cleverness ahoy!
//
// This works because it's tremendously unlikely for an entry to corrupt
// another while still preserving the string length of the JSON in
// question. So, we just slap the length in there and verify it on read.
//
// Thanks to @isaacs for the whiteboarding session that ended up with
// this.
await fs.appendFile(bucket, `\n${hashEntry(stringified)}\t${stringified}`)
await fixOwner.chownr(cache, bucket)
} catch (err) {
if (err.code === 'ENOENT') {
return undefined
}
throw err
// There's a class of race conditions that happen when things get deleted
// during fixOwner, or between the two mkdirfix/chownr calls.
//
// It's perfectly fine to just not bother in those cases and lie
// that the index entry was written. Because it's a cache.
}
return formatEntry(cache, entry)
}
module.exports.insert.sync = insertSync
function insertSync (cache, key, integrity, opts = {}) {
const { metadata, size } = opts
const bucket = bucketPath(cache, key)
const entry = {
key,
integrity: integrity && ssri.stringify(integrity),
time: Date.now(),
size,
metadata,
}
fixOwner.mkdirfix.sync(cache, path.dirname(bucket))
const stringified = JSON.stringify(entry)
fs.appendFileSync(bucket, `\n${hashEntry(stringified)}\t${stringified}`)
try {
fixOwner.chownr.sync(cache, bucket)
} catch (err) {
if (err.code !== 'ENOENT') {
throw err
}
}
return formatEntry(cache, entry)
}
module.exports.find = find
async function find (cache, key) {
const bucket = bucketPath(cache, key)
try {
const entries = await bucketEntries(bucket)
return entries.reduce((latest, next) => {
if (next && next.key === key) {
return formatEntry(cache, next)
} else {
return latest
}
}, null)
} catch (err) {
if (err.code === 'ENOENT') {
return null
} else {
throw err
}
}
}
module.exports.find.sync = findSync
function findSync (cache, key) {
const bucket = bucketPath(cache, key)
try {
return bucketEntriesSync(bucket).reduce((latest, next) => {
if (next && next.key === key) {
return formatEntry(cache, next)
} else {
return latest
}
}, null)
} catch (err) {
if (err.code === 'ENOENT') {
return null
} else {
throw err
}
}
}
module.exports.delete = del
function del (cache, key, opts = {}) {
if (!opts.removeFully) {
return insert(cache, key, null, opts)
}
const bucket = bucketPath(cache, key)
return rimraf(bucket)
}
module.exports.delete.sync = delSync
function delSync (cache, key, opts = {}) {
if (!opts.removeFully) {
return insertSync(cache, key, null, opts)
}
const bucket = bucketPath(cache, key)
return rimraf.sync(bucket)
}
module.exports.lsStream = lsStream
function lsStream (cache) {
const indexDir = bucketDir(cache)
const stream = new Minipass({ objectMode: true })
// Set all this up to run on the stream and then just return the stream
Promise.resolve().then(async () => {
const buckets = await readdirOrEmpty(indexDir)
await Promise.all(buckets.map(async (bucket) => {
const bucketPath = path.join(indexDir, bucket)
const subbuckets = await readdirOrEmpty(bucketPath)
await Promise.all(subbuckets.map(async (subbucket) => {
const subbucketPath = path.join(bucketPath, subbucket)
// "/cachename/<bucket 0xFF>/<bucket 0xFF>./*"
const subbucketEntries = await readdirOrEmpty(subbucketPath)
await Promise.all(subbucketEntries.map(async (entry) => {
const entryPath = path.join(subbucketPath, entry)
try {
const entries = await bucketEntries(entryPath)
// using a Map here prevents duplicate keys from showing up
// twice, I guess?
const reduced = entries.reduce((acc, entry) => {
acc.set(entry.key, entry)
return acc
}, new Map())
// reduced is a map of key => entry
for (const entry of reduced.values()) {
const formatted = formatEntry(cache, entry)
if (formatted) {
stream.write(formatted)
}
}
} catch (err) {
if (err.code === 'ENOENT') {
return undefined
}
throw err
}
}))
}))
}))
stream.end()
return stream
}).catch(err => stream.emit('error', err))
return stream
}
module.exports.ls = ls
async function ls (cache) {
const entries = await lsStream(cache).collect()
return entries.reduce((acc, xs) => {
acc[xs.key] = xs
return acc
}, {})
}
module.exports.bucketEntries = bucketEntries
async function bucketEntries (bucket, filter) {
const data = await fs.readFile(bucket, 'utf8')
return _bucketEntries(data, filter)
}
module.exports.bucketEntries.sync = bucketEntriesSync
function bucketEntriesSync (bucket, filter) {
const data = fs.readFileSync(bucket, 'utf8')
return _bucketEntries(data, filter)
}
function _bucketEntries (data, filter) {
const entries = []
data.split('\n').forEach((entry) => {
if (!entry) {
return
}
const pieces = entry.split('\t')
if (!pieces[1] || hashEntry(pieces[1]) !== pieces[0]) {
// Hash is no good! Corruption or malice? Doesn't matter!
// EJECT EJECT
return
}
let obj
try {
obj = JSON.parse(pieces[1])
} catch (e) {
// Entry is corrupted!
return
}
if (obj) {
entries.push(obj)
}
})
return entries
}
module.exports.bucketDir = bucketDir
function bucketDir (cache) {
return path.join(cache, `index-v${indexV}`)
}
module.exports.bucketPath = bucketPath
function bucketPath (cache, key) {
const hashed = hashKey(key)
return path.join.apply(
path,
[bucketDir(cache)].concat(hashToSegments(hashed))
)
}
module.exports.hashKey = hashKey
function hashKey (key) {
return hash(key, 'sha256')
}
module.exports.hashEntry = hashEntry
function hashEntry (str) {
return hash(str, 'sha1')
}
function hash (str, digest) {
return crypto
.createHash(digest)
.update(str)
.digest('hex')
}
function formatEntry (cache, entry, keepAll) {
// Treat null digests as deletions. They'll shadow any previous entries.
if (!entry.integrity && !keepAll) {
return null
}
return {
key: entry.key,
integrity: entry.integrity,
path: entry.integrity ? contentPath(cache, entry.integrity) : undefined,
size: entry.size,
time: entry.time,
metadata: entry.metadata,
}
}
function readdirOrEmpty (dir) {
return fs.readdir(dir).catch((err) => {
if (err.code === 'ENOENT' || err.code === 'ENOTDIR') {
return []
}
throw err
})
}
@@ -0,0 +1,225 @@
'use strict'
const Collect = require('minipass-collect')
const Minipass = require('minipass')
const Pipeline = require('minipass-pipeline')
const index = require('./entry-index')
const memo = require('./memoization')
const read = require('./content/read')
async function getData (cache, key, opts = {}) {
const { integrity, memoize, size } = opts
const memoized = memo.get(cache, key, opts)
if (memoized && memoize !== false) {
return {
metadata: memoized.entry.metadata,
data: memoized.data,
integrity: memoized.entry.integrity,
size: memoized.entry.size,
}
}
const entry = await index.find(cache, key, opts)
if (!entry) {
throw new index.NotFoundError(cache, key)
}
const data = await read(cache, entry.integrity, { integrity, size })
if (memoize) {
memo.put(cache, entry, data, opts)
}
return {
data,
metadata: entry.metadata,
size: entry.size,
integrity: entry.integrity,
}
}
module.exports = getData
async function getDataByDigest (cache, key, opts = {}) {
const { integrity, memoize, size } = opts
const memoized = memo.get.byDigest(cache, key, opts)
if (memoized && memoize !== false) {
return memoized
}
const res = await read(cache, key, { integrity, size })
if (memoize) {
memo.put.byDigest(cache, key, res, opts)
}
return res
}
module.exports.byDigest = getDataByDigest
function getDataSync (cache, key, opts = {}) {
const { integrity, memoize, size } = opts
const memoized = memo.get(cache, key, opts)
if (memoized && memoize !== false) {
return {
metadata: memoized.entry.metadata,
data: memoized.data,
integrity: memoized.entry.integrity,
size: memoized.entry.size,
}
}
const entry = index.find.sync(cache, key, opts)
if (!entry) {
throw new index.NotFoundError(cache, key)
}
const data = read.sync(cache, entry.integrity, {
integrity: integrity,
size: size,
})
const res = {
metadata: entry.metadata,
data: data,
size: entry.size,
integrity: entry.integrity,
}
if (memoize) {
memo.put(cache, entry, res.data, opts)
}
return res
}
module.exports.sync = getDataSync
function getDataByDigestSync (cache, digest, opts = {}) {
const { integrity, memoize, size } = opts
const memoized = memo.get.byDigest(cache, digest, opts)
if (memoized && memoize !== false) {
return memoized
}
const res = read.sync(cache, digest, {
integrity: integrity,
size: size,
})
if (memoize) {
memo.put.byDigest(cache, digest, res, opts)
}
return res
}
module.exports.sync.byDigest = getDataByDigestSync
const getMemoizedStream = (memoized) => {
const stream = new Minipass()
stream.on('newListener', function (ev, cb) {
ev === 'metadata' && cb(memoized.entry.metadata)
ev === 'integrity' && cb(memoized.entry.integrity)
ev === 'size' && cb(memoized.entry.size)
})
stream.end(memoized.data)
return stream
}
function getStream (cache, key, opts = {}) {
const { memoize, size } = opts
const memoized = memo.get(cache, key, opts)
if (memoized && memoize !== false) {
return getMemoizedStream(memoized)
}
const stream = new Pipeline()
// Set all this up to run on the stream and then just return the stream
Promise.resolve().then(async () => {
const entry = await index.find(cache, key)
if (!entry) {
throw new index.NotFoundError(cache, key)
}
stream.emit('metadata', entry.metadata)
stream.emit('integrity', entry.integrity)
stream.emit('size', entry.size)
stream.on('newListener', function (ev, cb) {
ev === 'metadata' && cb(entry.metadata)
ev === 'integrity' && cb(entry.integrity)
ev === 'size' && cb(entry.size)
})
const src = read.readStream(
cache,
entry.integrity,
{ ...opts, size: typeof size !== 'number' ? entry.size : size }
)
if (memoize) {
const memoStream = new Collect.PassThrough()
memoStream.on('collect', data => memo.put(cache, entry, data, opts))
stream.unshift(memoStream)
}
stream.unshift(src)
return stream
}).catch((err) => stream.emit('error', err))
return stream
}
module.exports.stream = getStream
function getStreamDigest (cache, integrity, opts = {}) {
const { memoize } = opts
const memoized = memo.get.byDigest(cache, integrity, opts)
if (memoized && memoize !== false) {
const stream = new Minipass()
stream.end(memoized)
return stream
} else {
const stream = read.readStream(cache, integrity, opts)
if (!memoize) {
return stream
}
const memoStream = new Collect.PassThrough()
memoStream.on('collect', data => memo.put.byDigest(
cache,
integrity,
data,
opts
))
return new Pipeline(stream, memoStream)
}
}
module.exports.stream.byDigest = getStreamDigest
function info (cache, key, opts = {}) {
const { memoize } = opts
const memoized = memo.get(cache, key, opts)
if (memoized && memoize !== false) {
return Promise.resolve(memoized.entry)
} else {
return index.find(cache, key)
}
}
module.exports.info = info
async function copy (cache, key, dest, opts = {}) {
const entry = await index.find(cache, key, opts)
if (!entry) {
throw new index.NotFoundError(cache, key)
}
await read.copy(cache, entry.integrity, dest, opts)
return {
metadata: entry.metadata,
size: entry.size,
integrity: entry.integrity,
}
}
module.exports.copy = copy
async function copyByDigest (cache, key, dest, opts = {}) {
await read.copy(cache, key, dest, opts)
return key
}
module.exports.copy.byDigest = copyByDigest
module.exports.hasContent = read.hasContent
@@ -0,0 +1,45 @@
'use strict'
const get = require('./get.js')
const put = require('./put.js')
const rm = require('./rm.js')
const verify = require('./verify.js')
const { clearMemoized } = require('./memoization.js')
const tmp = require('./util/tmp.js')
const index = require('./entry-index.js')
module.exports.index = {}
module.exports.index.compact = index.compact
module.exports.index.insert = index.insert
module.exports.ls = index.ls
module.exports.ls.stream = index.lsStream
module.exports.get = get
module.exports.get.byDigest = get.byDigest
module.exports.get.sync = get.sync
module.exports.get.sync.byDigest = get.sync.byDigest
module.exports.get.stream = get.stream
module.exports.get.stream.byDigest = get.stream.byDigest
module.exports.get.copy = get.copy
module.exports.get.copy.byDigest = get.copy.byDigest
module.exports.get.info = get.info
module.exports.get.hasContent = get.hasContent
module.exports.get.hasContent.sync = get.hasContent.sync
module.exports.put = put
module.exports.put.stream = put.stream
module.exports.rm = rm.entry
module.exports.rm.all = rm.all
module.exports.rm.entry = module.exports.rm
module.exports.rm.content = rm.content
module.exports.clearMemoized = clearMemoized
module.exports.tmp = {}
module.exports.tmp.mkdir = tmp.mkdir
module.exports.tmp.withTmp = tmp.withTmp
module.exports.verify = verify
module.exports.verify.lastRun = verify.lastRun
@@ -0,0 +1,72 @@
'use strict'
const LRU = require('lru-cache')
const MEMOIZED = new LRU({
max: 500,
maxSize: 50 * 1024 * 1024, // 50MB
ttl: 3 * 60 * 1000, // 3 minutes
sizeCalculation: (entry, key) => key.startsWith('key:') ? entry.data.length : entry.length,
})
module.exports.clearMemoized = clearMemoized
function clearMemoized () {
const old = {}
MEMOIZED.forEach((v, k) => {
old[k] = v
})
MEMOIZED.clear()
return old
}
module.exports.put = put
function put (cache, entry, data, opts) {
pickMem(opts).set(`key:${cache}:${entry.key}`, { entry, data })
putDigest(cache, entry.integrity, data, opts)
}
module.exports.put.byDigest = putDigest
function putDigest (cache, integrity, data, opts) {
pickMem(opts).set(`digest:${cache}:${integrity}`, data)
}
module.exports.get = get
function get (cache, key, opts) {
return pickMem(opts).get(`key:${cache}:${key}`)
}
module.exports.get.byDigest = getDigest
function getDigest (cache, integrity, opts) {
return pickMem(opts).get(`digest:${cache}:${integrity}`)
}
class ObjProxy {
constructor (obj) {
this.obj = obj
}
get (key) {
return this.obj[key]
}
set (key, val) {
this.obj[key] = val
}
}
function pickMem (opts) {
if (!opts || !opts.memoize) {
return MEMOIZED
} else if (opts.memoize.get && opts.memoize.set) {
return opts.memoize
} else if (typeof opts.memoize === 'object') {
return new ObjProxy(opts.memoize)
} else {
return MEMOIZED
}
}
@@ -0,0 +1,80 @@
'use strict'
const index = require('./entry-index')
const memo = require('./memoization')
const write = require('./content/write')
const Flush = require('minipass-flush')
const { PassThrough } = require('minipass-collect')
const Pipeline = require('minipass-pipeline')
const putOpts = (opts) => ({
algorithms: ['sha512'],
...opts,
})
module.exports = putData
async function putData (cache, key, data, opts = {}) {
const { memoize } = opts
opts = putOpts(opts)
const res = await write(cache, data, opts)
const entry = await index.insert(cache, key, res.integrity, { ...opts, size: res.size })
if (memoize) {
memo.put(cache, entry, data, opts)
}
return res.integrity
}
module.exports.stream = putStream
function putStream (cache, key, opts = {}) {
const { memoize } = opts
opts = putOpts(opts)
let integrity
let size
let error
let memoData
const pipeline = new Pipeline()
// first item in the pipeline is the memoizer, because we need
// that to end first and get the collected data.
if (memoize) {
const memoizer = new PassThrough().on('collect', data => {
memoData = data
})
pipeline.push(memoizer)
}
// contentStream is a write-only, not a passthrough
// no data comes out of it.
const contentStream = write.stream(cache, opts)
.on('integrity', (int) => {
integrity = int
})
.on('size', (s) => {
size = s
})
.on('error', (err) => {
error = err
})
pipeline.push(contentStream)
// last but not least, we write the index and emit hash and size,
// and memoize if we're doing that
pipeline.push(new Flush({
async flush () {
if (!error) {
const entry = await index.insert(cache, key, integrity, { ...opts, size })
if (memoize && memoData) {
memo.put(cache, entry, memoData, opts)
}
pipeline.emit('integrity', integrity)
pipeline.emit('size', size)
}
},
}))
return pipeline
}
@@ -0,0 +1,31 @@
'use strict'
const util = require('util')
const index = require('./entry-index')
const memo = require('./memoization')
const path = require('path')
const rimraf = util.promisify(require('rimraf'))
const rmContent = require('./content/rm')
module.exports = entry
module.exports.entry = entry
function entry (cache, key, opts) {
memo.clearMemoized()
return index.delete(cache, key, opts)
}
module.exports.content = content
function content (cache, integrity) {
memo.clearMemoized()
return rmContent(cache, integrity)
}
module.exports.all = all
function all (cache) {
memo.clearMemoized()
return rimraf(path.join(cache, '*(content-*|index-*)'))
}
@@ -0,0 +1,145 @@
'use strict'
const util = require('util')
const chownr = util.promisify(require('chownr'))
const mkdirp = require('mkdirp')
const inflight = require('promise-inflight')
const inferOwner = require('infer-owner')
// Memoize getuid()/getgid() calls.
// patch process.setuid/setgid to invalidate cached value on change
const self = { uid: null, gid: null }
const getSelf = () => {
if (typeof self.uid !== 'number') {
self.uid = process.getuid()
const setuid = process.setuid
process.setuid = (uid) => {
self.uid = null
process.setuid = setuid
return process.setuid(uid)
}
}
if (typeof self.gid !== 'number') {
self.gid = process.getgid()
const setgid = process.setgid
process.setgid = (gid) => {
self.gid = null
process.setgid = setgid
return process.setgid(gid)
}
}
}
module.exports.chownr = fixOwner
async function fixOwner (cache, filepath) {
if (!process.getuid) {
// This platform doesn't need ownership fixing
return
}
getSelf()
if (self.uid !== 0) {
// almost certainly can't chown anyway
return
}
const { uid, gid } = await inferOwner(cache)
// No need to override if it's already what we used.
if (self.uid === uid && self.gid === gid) {
return
}
return inflight('fixOwner: fixing ownership on ' + filepath, () =>
chownr(
filepath,
typeof uid === 'number' ? uid : self.uid,
typeof gid === 'number' ? gid : self.gid
).catch((err) => {
if (err.code === 'ENOENT') {
return null
}
throw err
})
)
}
module.exports.chownr.sync = fixOwnerSync
function fixOwnerSync (cache, filepath) {
if (!process.getuid) {
// This platform doesn't need ownership fixing
return
}
const { uid, gid } = inferOwner.sync(cache)
getSelf()
if (self.uid !== 0) {
// almost certainly can't chown anyway
return
}
if (self.uid === uid && self.gid === gid) {
// No need to override if it's already what we used.
return
}
try {
chownr.sync(
filepath,
typeof uid === 'number' ? uid : self.uid,
typeof gid === 'number' ? gid : self.gid
)
} catch (err) {
// only catch ENOENT, any other error is a problem.
if (err.code === 'ENOENT') {
return null
}
throw err
}
}
module.exports.mkdirfix = mkdirfix
async function mkdirfix (cache, p, cb) {
// we have to infer the owner _before_ making the directory, even though
// we aren't going to use the results, since the cache itself might not
// exist yet. If we mkdirp it, then our current uid/gid will be assumed
// to be correct if it creates the cache folder in the process.
await inferOwner(cache)
try {
const made = await mkdirp(p)
if (made) {
await fixOwner(cache, made)
return made
}
} catch (err) {
if (err.code === 'EEXIST') {
await fixOwner(cache, p)
return null
}
throw err
}
}
module.exports.mkdirfix.sync = mkdirfixSync
function mkdirfixSync (cache, p) {
try {
inferOwner.sync(cache)
const made = mkdirp.sync(p)
if (made) {
fixOwnerSync(cache, made)
return made
}
} catch (err) {
if (err.code === 'EEXIST') {
fixOwnerSync(cache, p)
return null
} else {
throw err
}
}
}
@@ -0,0 +1,7 @@
'use strict'
module.exports = hashToSegments
function hashToSegments (hash) {
return [hash.slice(0, 2), hash.slice(2, 4), hash.slice(4)]
}
@@ -0,0 +1,56 @@
'use strict'
const fs = require('@npmcli/fs')
const move = require('@npmcli/move-file')
const pinflight = require('promise-inflight')
module.exports = moveFile
async function moveFile (src, dest) {
const isWindows = process.platform === 'win32'
// This isn't quite an fs.rename -- the assumption is that
// if `dest` already exists, and we get certain errors while
// trying to move it, we should just not bother.
//
// In the case of cache corruption, users will receive an
// EINTEGRITY error elsewhere, and can remove the offending
// content their own way.
//
// Note that, as the name suggests, this strictly only supports file moves.
try {
await fs.link(src, dest)
} catch (err) {
if (isWindows && err.code === 'EPERM') {
// XXX This is a really weird way to handle this situation, as it
// results in the src file being deleted even though the dest
// might not exist. Since we pretty much always write files to
// deterministic locations based on content hash, this is likely
// ok (or at worst, just ends in a future cache miss). But it would
// be worth investigating at some time in the future if this is
// really what we want to do here.
} else if (err.code === 'EEXIST' || err.code === 'EBUSY') {
// file already exists, so whatever
} else {
throw err
}
}
try {
await Promise.all([
fs.unlink(src),
!isWindows && fs.chmod(dest, '0444'),
])
} catch (e) {
return pinflight('cacache-move-file:' + dest, async () => {
await fs.stat(dest).catch((err) => {
if (err.code !== 'ENOENT') {
// Something else is wrong here. Bail bail bail
throw err
}
})
// file doesn't already exist! let's try a rename -> copy fallback
// only delete if it successfully copies
return move(src, dest)
})
}
}
@@ -0,0 +1,33 @@
'use strict'
const fs = require('@npmcli/fs')
const fixOwner = require('./fix-owner')
const path = require('path')
module.exports.mkdir = mktmpdir
async function mktmpdir (cache, opts = {}) {
const { tmpPrefix } = opts
const tmpDir = path.join(cache, 'tmp')
await fs.mkdir(tmpDir, { recursive: true, owner: 'inherit' })
// do not use path.join(), it drops the trailing / if tmpPrefix is unset
const target = `${tmpDir}${path.sep}${tmpPrefix || ''}`
return fs.mkdtemp(target, { owner: 'inherit' })
}
module.exports.withTmp = withTmp
function withTmp (cache, opts, cb) {
if (!cb) {
cb = opts
opts = {}
}
return fs.withTempDir(path.join(cache, 'tmp'), cb, opts)
}
module.exports.fix = fixtmpdir
function fixtmpdir (cache) {
return fixOwner(cache, path.join(cache, 'tmp'))
}
@@ -0,0 +1,257 @@
'use strict'
const util = require('util')
const pMap = require('p-map')
const contentPath = require('./content/path')
const fixOwner = require('./util/fix-owner')
const fs = require('@npmcli/fs')
const fsm = require('fs-minipass')
const glob = util.promisify(require('glob'))
const index = require('./entry-index')
const path = require('path')
const rimraf = util.promisify(require('rimraf'))
const ssri = require('ssri')
const globify = pattern => pattern.split('\\').join('/')
const hasOwnProperty = (obj, key) =>
Object.prototype.hasOwnProperty.call(obj, key)
const verifyOpts = (opts) => ({
concurrency: 20,
log: { silly () {} },
...opts,
})
module.exports = verify
async function verify (cache, opts) {
opts = verifyOpts(opts)
opts.log.silly('verify', 'verifying cache at', cache)
const steps = [
markStartTime,
fixPerms,
garbageCollect,
rebuildIndex,
cleanTmp,
writeVerifile,
markEndTime,
]
const stats = {}
for (const step of steps) {
const label = step.name
const start = new Date()
const s = await step(cache, opts)
if (s) {
Object.keys(s).forEach((k) => {
stats[k] = s[k]
})
}
const end = new Date()
if (!stats.runTime) {
stats.runTime = {}
}
stats.runTime[label] = end - start
}
stats.runTime.total = stats.endTime - stats.startTime
opts.log.silly(
'verify',
'verification finished for',
cache,
'in',
`${stats.runTime.total}ms`
)
return stats
}
async function markStartTime (cache, opts) {
return { startTime: new Date() }
}
async function markEndTime (cache, opts) {
return { endTime: new Date() }
}
async function fixPerms (cache, opts) {
opts.log.silly('verify', 'fixing cache permissions')
await fixOwner.mkdirfix(cache, cache)
// TODO - fix file permissions too
await fixOwner.chownr(cache, cache)
return null
}
// Implements a naive mark-and-sweep tracing garbage collector.
//
// The algorithm is basically as follows:
// 1. Read (and filter) all index entries ("pointers")
// 2. Mark each integrity value as "live"
// 3. Read entire filesystem tree in `content-vX/` dir
// 4. If content is live, verify its checksum and delete it if it fails
// 5. If content is not marked as live, rimraf it.
//
async function garbageCollect (cache, opts) {
opts.log.silly('verify', 'garbage collecting content')
const indexStream = index.lsStream(cache)
const liveContent = new Set()
indexStream.on('data', (entry) => {
if (opts.filter && !opts.filter(entry)) {
return
}
liveContent.add(entry.integrity.toString())
})
await new Promise((resolve, reject) => {
indexStream.on('end', resolve).on('error', reject)
})
const contentDir = contentPath.contentDir(cache)
const files = await glob(globify(path.join(contentDir, '**')), {
follow: false,
nodir: true,
nosort: true,
})
const stats = {
verifiedContent: 0,
reclaimedCount: 0,
reclaimedSize: 0,
badContentCount: 0,
keptSize: 0,
}
await pMap(
files,
async (f) => {
const split = f.split(/[/\\]/)
const digest = split.slice(split.length - 3).join('')
const algo = split[split.length - 4]
const integrity = ssri.fromHex(digest, algo)
if (liveContent.has(integrity.toString())) {
const info = await verifyContent(f, integrity)
if (!info.valid) {
stats.reclaimedCount++
stats.badContentCount++
stats.reclaimedSize += info.size
} else {
stats.verifiedContent++
stats.keptSize += info.size
}
} else {
// No entries refer to this content. We can delete.
stats.reclaimedCount++
const s = await fs.stat(f)
await rimraf(f)
stats.reclaimedSize += s.size
}
return stats
},
{ concurrency: opts.concurrency }
)
return stats
}
async function verifyContent (filepath, sri) {
const contentInfo = {}
try {
const { size } = await fs.stat(filepath)
contentInfo.size = size
contentInfo.valid = true
await ssri.checkStream(new fsm.ReadStream(filepath), sri)
} catch (err) {
if (err.code === 'ENOENT') {
return { size: 0, valid: false }
}
if (err.code !== 'EINTEGRITY') {
throw err
}
await rimraf(filepath)
contentInfo.valid = false
}
return contentInfo
}
async function rebuildIndex (cache, opts) {
opts.log.silly('verify', 'rebuilding index')
const entries = await index.ls(cache)
const stats = {
missingContent: 0,
rejectedEntries: 0,
totalEntries: 0,
}
const buckets = {}
for (const k in entries) {
/* istanbul ignore else */
if (hasOwnProperty(entries, k)) {
const hashed = index.hashKey(k)
const entry = entries[k]
const excluded = opts.filter && !opts.filter(entry)
excluded && stats.rejectedEntries++
if (buckets[hashed] && !excluded) {
buckets[hashed].push(entry)
} else if (buckets[hashed] && excluded) {
// skip
} else if (excluded) {
buckets[hashed] = []
buckets[hashed]._path = index.bucketPath(cache, k)
} else {
buckets[hashed] = [entry]
buckets[hashed]._path = index.bucketPath(cache, k)
}
}
}
await pMap(
Object.keys(buckets),
(key) => {
return rebuildBucket(cache, buckets[key], stats, opts)
},
{ concurrency: opts.concurrency }
)
return stats
}
async function rebuildBucket (cache, bucket, stats, opts) {
await fs.truncate(bucket._path)
// This needs to be serialized because cacache explicitly
// lets very racy bucket conflicts clobber each other.
for (const entry of bucket) {
const content = contentPath(cache, entry.integrity)
try {
await fs.stat(content)
await index.insert(cache, entry.key, entry.integrity, {
metadata: entry.metadata,
size: entry.size,
})
stats.totalEntries++
} catch (err) {
if (err.code === 'ENOENT') {
stats.rejectedEntries++
stats.missingContent++
} else {
throw err
}
}
}
}
function cleanTmp (cache, opts) {
opts.log.silly('verify', 'cleaning tmp directory')
return rimraf(path.join(cache, 'tmp'))
}
function writeVerifile (cache, opts) {
const verifile = path.join(cache, '_lastverified')
opts.log.silly('verify', 'writing verifile to ' + verifile)
try {
return fs.writeFile(verifile, `${Date.now()}`)
} finally {
fixOwner.chownr.sync(cache, verifile)
}
}
module.exports.lastRun = lastRun
async function lastRun (cache) {
const data = await fs.readFile(path.join(cache, '_lastverified'), { encoding: 'utf8' })
return new Date(+data)
}
@@ -0,0 +1,84 @@
{
"name": "cacache",
"version": "16.1.3",
"cache-version": {
"content": "2",
"index": "5"
},
"description": "Fast, fault-tolerant, cross-platform, disk-based, data-agnostic, content-addressable cache.",
"main": "lib/index.js",
"files": [
"bin/",
"lib/"
],
"scripts": {
"preversion": "npm test",
"postversion": "npm publish",
"prepublishOnly": "git push origin --follow-tags",
"test": "tap",
"snap": "tap",
"coverage": "tap",
"test-docker": "docker run -it --rm --name pacotest -v \"$PWD\":/tmp -w /tmp node:latest npm test",
"lint": "eslint \"**/*.js\"",
"npmclilint": "npmcli-lint",
"lintfix": "npm run lint -- --fix",
"postsnap": "npm run lintfix --",
"postlint": "template-oss-check",
"posttest": "npm run lint",
"template-oss-apply": "template-oss-apply --force"
},
"repository": {
"type": "git",
"url": "https://github.com/npm/cacache.git"
},
"keywords": [
"cache",
"caching",
"content-addressable",
"sri",
"sri hash",
"subresource integrity",
"cache",
"storage",
"store",
"file store",
"filesystem",
"disk cache",
"disk storage"
],
"license": "ISC",
"dependencies": {
"@npmcli/fs": "^2.1.0",
"@npmcli/move-file": "^2.0.0",
"chownr": "^2.0.0",
"fs-minipass": "^2.1.0",
"glob": "^8.0.1",
"infer-owner": "^1.0.4",
"lru-cache": "^7.7.1",
"minipass": "^3.1.6",
"minipass-collect": "^1.0.2",
"minipass-flush": "^1.0.5",
"minipass-pipeline": "^1.2.4",
"mkdirp": "^1.0.4",
"p-map": "^4.0.0",
"promise-inflight": "^1.0.1",
"rimraf": "^3.0.2",
"ssri": "^9.0.0",
"tar": "^6.1.11",
"unique-filename": "^2.0.0"
},
"devDependencies": {
"@npmcli/eslint-config": "^3.0.1",
"@npmcli/template-oss": "3.5.0",
"tap": "^16.0.0"
},
"engines": {
"node": "^12.13.0 || ^14.15.0 || >=16.0.0"
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"windowsCI": false,
"version": "3.5.0"
},
"author": "GitHub Inc."
}
+16
View File
@@ -0,0 +1,16 @@
ISC License
Copyright 2021 (c) npm, Inc.
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE COPYRIGHT HOLDER DISCLAIMS
ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE
USE OR PERFORMANCE OF THIS SOFTWARE.
+528
View File
@@ -0,0 +1,528 @@
# ssri [![npm version](https://img.shields.io/npm/v/ssri.svg)](https://npm.im/ssri) [![license](https://img.shields.io/npm/l/ssri.svg)](https://npm.im/ssri) [![Travis](https://img.shields.io/travis/npm/ssri.svg)](https://travis-ci.org/npm/ssri) [![AppVeyor](https://ci.appveyor.com/api/projects/status/github/npm/ssri?svg=true)](https://ci.appveyor.com/project/npm/ssri) [![Coverage Status](https://coveralls.io/repos/github/npm/ssri/badge.svg?branch=latest)](https://coveralls.io/github/npm/ssri?branch=latest)
[`ssri`](https://github.com/npm/ssri), short for Standard Subresource
Integrity, is a Node.js utility for parsing, manipulating, serializing,
generating, and verifying [Subresource
Integrity](https://w3c.github.io/webappsec/specs/subresourceintegrity/) hashes.
## Install
`$ npm install --save ssri`
## Table of Contents
* [Example](#example)
* [Features](#features)
* [Contributing](#contributing)
* [API](#api)
* Parsing & Serializing
* [`parse`](#parse)
* [`stringify`](#stringify)
* [`Integrity#concat`](#integrity-concat)
* [`Integrity#merge`](#integrity-merge)
* [`Integrity#toString`](#integrity-to-string)
* [`Integrity#toJSON`](#integrity-to-json)
* [`Integrity#match`](#integrity-match)
* [`Integrity#pickAlgorithm`](#integrity-pick-algorithm)
* [`Integrity#hexDigest`](#integrity-hex-digest)
* Integrity Generation
* [`fromHex`](#from-hex)
* [`fromData`](#from-data)
* [`fromStream`](#from-stream)
* [`create`](#create)
* Integrity Verification
* [`checkData`](#check-data)
* [`checkStream`](#check-stream)
* [`integrityStream`](#integrity-stream)
### Example
```javascript
const ssri = require('ssri')
const integrity = 'sha512-9KhgCRIx/AmzC8xqYJTZRrnO8OW2Pxyl2DIMZSBOr0oDvtEFyht3xpp71j/r/pAe1DM+JI/A+line3jUBgzQ7A==?foo'
// Parsing and serializing
const parsed = ssri.parse(integrity)
ssri.stringify(parsed) // === integrity (works on non-Integrity objects)
parsed.toString() // === integrity
// Async stream functions
ssri.checkStream(fs.createReadStream('./my-file'), integrity).then(...)
ssri.fromStream(fs.createReadStream('./my-file')).then(sri => {
sri.toString() === integrity
})
fs.createReadStream('./my-file').pipe(ssri.createCheckerStream(sri))
// Sync data functions
ssri.fromData(fs.readFileSync('./my-file')) // === parsed
ssri.checkData(fs.readFileSync('./my-file'), integrity) // => 'sha512'
```
### Features
* Parses and stringifies SRI strings.
* Generates SRI strings from raw data or Streams.
* Strict standard compliance.
* `?foo` metadata option support.
* Multiple entries for the same algorithm.
* Object-based integrity hash manipulation.
* Small footprint: no dependencies, concise implementation.
* Full test coverage.
* Customizable algorithm picker.
### Contributing
The ssri team enthusiastically welcomes contributions and project participation!
There's a bunch of things you can do if you want to contribute! The [Contributor
Guide](CONTRIBUTING.md) has all the information you need for everything from
reporting bugs to contributing entire new features. Please don't hesitate to
jump in if you'd like to, or even ask us questions if something isn't clear.
### API
#### <a name="parse"></a> `> ssri.parse(sri, [opts]) -> Integrity`
Parses `sri` into an `Integrity` data structure. `sri` can be an integrity
string, an `Hash`-like with `digest` and `algorithm` fields and an optional
`options` field, or an `Integrity`-like object. The resulting object will be an
`Integrity` instance that has this shape:
```javascript
{
'sha1': [{algorithm: 'sha1', digest: 'deadbeef', options: []}],
'sha512': [
{algorithm: 'sha512', digest: 'c0ffee', options: []},
{algorithm: 'sha512', digest: 'bad1dea', options: ['foo']}
],
}
```
If `opts.single` is truthy, a single `Hash` object will be returned. That is, a
single object that looks like `{algorithm, digest, options}`, as opposed to a
larger object with multiple of these.
If `opts.strict` is truthy, the resulting object will be filtered such that
it strictly follows the Subresource Integrity spec, throwing away any entries
with any invalid components. This also means a restricted set of algorithms
will be used -- the spec limits them to `sha256`, `sha384`, and `sha512`.
Strict mode is recommended if the integrity strings are intended for use in
browsers, or in other situations where strict adherence to the spec is needed.
##### Example
```javascript
ssri.parse('sha512-9KhgCRIx/AmzC8xqYJTZRrnO8OW2Pxyl2DIMZSBOr0oDvtEFyht3xpp71j/r/pAe1DM+JI/A+line3jUBgzQ7A==?foo') // -> Integrity object
```
#### <a name="stringify"></a> `> ssri.stringify(sri, [opts]) -> String`
This function is identical to [`Integrity#toString()`](#integrity-to-string),
except it can be used on _any_ object that [`parse`](#parse) can handle -- that
is, a string, an `Hash`-like, or an `Integrity`-like.
The `opts.sep` option defines the string to use when joining multiple entries
together. To be spec-compliant, this _must_ be whitespace. The default is a
single space (`' '`).
If `opts.strict` is true, the integrity string will be created using strict
parsing rules. See [`ssri.parse`](#parse).
##### Example
```javascript
// Useful for cleaning up input SRI strings:
ssri.stringify('\n\rsha512-foo\n\t\tsha384-bar')
// -> 'sha512-foo sha384-bar'
// Hash-like: only a single entry.
ssri.stringify({
algorithm: 'sha512',
digest:'9KhgCRIx/AmzC8xqYJTZRrnO8OW2Pxyl2DIMZSBOr0oDvtEFyht3xpp71j/r/pAe1DM+JI/A+line3jUBgzQ7A==',
options: ['foo']
})
// ->
// 'sha512-9KhgCRIx/AmzC8xqYJTZRrnO8OW2Pxyl2DIMZSBOr0oDvtEFyht3xpp71j/r/pAe1DM+JI/A+line3jUBgzQ7A==?foo'
// Integrity-like: full multi-entry syntax. Similar to output of `ssri.parse`
ssri.stringify({
'sha512': [
{
algorithm: 'sha512',
digest:'9KhgCRIx/AmzC8xqYJTZRrnO8OW2Pxyl2DIMZSBOr0oDvtEFyht3xpp71j/r/pAe1DM+JI/A+line3jUBgzQ7A==',
options: ['foo']
}
]
})
// ->
// 'sha512-9KhgCRIx/AmzC8xqYJTZRrnO8OW2Pxyl2DIMZSBOr0oDvtEFyht3xpp71j/r/pAe1DM+JI/A+line3jUBgzQ7A==?foo'
```
#### <a name="integrity-concat"></a> `> Integrity#concat(otherIntegrity, [opts]) -> Integrity`
Concatenates an `Integrity` object with another IntegrityLike, or an integrity
string.
This is functionally equivalent to concatenating the string format of both
integrity arguments, and calling [`ssri.parse`](#ssri-parse) on the new string.
If `opts.strict` is true, the new `Integrity` will be created using strict
parsing rules. See [`ssri.parse`](#parse).
##### Example
```javascript
// This will combine the integrity checks for two different versions of
// your index.js file so you can use a single integrity string and serve
// either of these to clients, from a single `<script>` tag.
const desktopIntegrity = ssri.fromData(fs.readFileSync('./index.desktop.js'))
const mobileIntegrity = ssri.fromData(fs.readFileSync('./index.mobile.js'))
// Note that browsers (and ssri) will succeed as long as ONE of the entries
// for the *prioritized* algorithm succeeds. That is, in order for this fallback
// to work, both desktop and mobile *must* use the same `algorithm` values.
desktopIntegrity.concat(mobileIntegrity)
```
#### <a name="integrity-merge"></a> `> Integrity#merge(otherIntegrity, [opts])`
Safely merges another IntegrityLike or integrity string into an `Integrity`
object.
If the other integrity value has any algorithms in common with the current
object, then the hash digests must match, or an error is thrown.
Any new hashes will be added to the current object's set.
This is useful when an integrity value may be upgraded with a stronger
algorithm, you wish to prevent accidentally suppressing integrity errors by
overwriting the expected integrity value.
##### Example
```javascript
const data = fs.readFileSync('data.txt')
// integrity.txt contains 'sha1-X1UT+IIv2+UUWvM7ZNjZcNz5XG4='
// because we were young, and didn't realize sha1 would not last
const expectedIntegrity = ssri.parse(fs.readFileSync('integrity.txt', 'utf8'))
const match = ssri.checkData(data, expectedIntegrity, {
algorithms: ['sha512', 'sha1']
})
if (!match) {
throw new Error('data corrupted or something!')
}
// get a stronger algo!
if (match && match.algorithm !== 'sha512') {
const updatedIntegrity = ssri.fromData(data, { algorithms: ['sha512'] })
expectedIntegrity.merge(updatedIntegrity)
fs.writeFileSync('integrity.txt', expectedIntegrity.toString())
// file now contains
// 'sha1-X1UT+IIv2+UUWvM7ZNjZcNz5XG4= sha512-yzd8ELD1piyANiWnmdnpCL5F52f10UfUdEkHywVZeqTt0ymgrxR63Qz0GB7TKPoeeZQmWCaz7T1+9vBnypkYWg=='
}
```
#### <a name="integrity-to-string"></a> `> Integrity#toString([opts]) -> String`
Returns the string representation of an `Integrity` object. All hash entries
will be concatenated in the string by `opts.sep`, which defaults to `' '`.
If you want to serialize an object that didn't come from an `ssri` function,
use [`ssri.stringify()`](#stringify).
If `opts.strict` is true, the integrity string will be created using strict
parsing rules. See [`ssri.parse`](#parse).
##### Example
```javascript
const integrity = 'sha512-9KhgCRIx/AmzC8xqYJTZRrnO8OW2Pxyl2DIMZSBOr0oDvtEFyht3xpp71j/r/pAe1DM+JI/A+line3jUBgzQ7A==?foo'
ssri.parse(integrity).toString() === integrity
```
#### <a name="integrity-to-json"></a> `> Integrity#toJSON() -> String`
Returns the string representation of an `Integrity` object. All hash entries
will be concatenated in the string by `' '`.
This is a convenience method so you can pass an `Integrity` object directly to `JSON.stringify`.
For more info check out [toJSON() behavior on mdn](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify#toJSON%28%29_behavior).
##### Example
```javascript
const integrity = '"sha512-9KhgCRIx/AmzC8xqYJTZRrnO8OW2Pxyl2DIMZSBOr0oDvtEFyht3xpp71j/r/pAe1DM+JI/A+line3jUBgzQ7A==?foo"'
JSON.stringify(ssri.parse(integrity)) === integrity
```
#### <a name="integrity-match"></a> `> Integrity#match(sri, [opts]) -> Hash | false`
Returns the matching (truthy) hash if `Integrity` matches the argument passed as
`sri`, which can be anything that [`parse`](#parse) will accept. `opts` will be
passed through to `parse` and [`pickAlgorithm()`](#integrity-pick-algorithm).
##### Example
```javascript
const integrity = 'sha512-9KhgCRIx/AmzC8xqYJTZRrnO8OW2Pxyl2DIMZSBOr0oDvtEFyht3xpp71j/r/pAe1DM+JI/A+line3jUBgzQ7A=='
ssri.parse(integrity).match(integrity)
// Hash {
// digest: '9KhgCRIx/AmzC8xqYJTZRrnO8OW2Pxyl2DIMZSBOr0oDvtEFyht3xpp71j/r/pAe1DM+JI/A+line3jUBgzQ7A=='
// algorithm: 'sha512'
// }
ssri.parse(integrity).match('sha1-deadbeef')
// false
```
#### <a name="integrity-pick-algorithm"></a> `> Integrity#pickAlgorithm([opts]) -> String`
Returns the "best" algorithm from those available in the integrity object.
If `opts.pickAlgorithm` is provided, it will be passed two algorithms as
arguments. ssri will prioritize whichever of the two algorithms is returned by
this function. Note that the function may be called multiple times, and it
**must** return one of the two algorithms provided. By default, ssri will make
a best-effort to pick the strongest/most reliable of the given algorithms. It
may intentionally deprioritize algorithms with known vulnerabilities.
##### Example
```javascript
ssri.parse('sha1-WEakDigEST sha512-yzd8ELD1piyANiWnmdnpCL5F52f10UfUdEkHywVZeqTt0ymgrxR63Qz0GB7TKPoeeZQmWCaz7T1').pickAlgorithm() // sha512
```
#### <a name="integrity-hex-digest"></a> `> Integrity#hexDigest() -> String`
`Integrity` is assumed to be either a single-hash `Integrity` instance, or a
`Hash` instance. Returns its `digest`, converted to a hex representation of the
base64 data.
##### Example
```javascript
ssri.parse('sha1-deadbeef').hexDigest() // '75e69d6de79f'
```
#### <a name="from-hex"></a> `> ssri.fromHex(hexDigest, algorithm, [opts]) -> Integrity`
Creates an `Integrity` object with a single entry, based on a hex-formatted
hash. This is a utility function to help convert existing shasums to the
Integrity format, and is roughly equivalent to something like:
```javascript
algorithm + '-' + Buffer.from(hexDigest, 'hex').toString('base64')
```
`opts.options` may optionally be passed in: it must be an array of option
strings that will be added to all generated integrity hashes generated by
`fromData`. This is a loosely-specified feature of SRIs, and currently has no
specified semantics besides being `?`-separated. Use at your own risk, and
probably avoid if your integrity strings are meant to be used with browsers.
If `opts.strict` is true, the integrity object will be created using strict
parsing rules. See [`ssri.parse`](#parse).
If `opts.single` is true, a single `Hash` object will be returned.
##### Example
```javascript
ssri.fromHex('75e69d6de79f', 'sha1').toString() // 'sha1-deadbeef'
```
#### <a name="from-data"></a> `> ssri.fromData(data, [opts]) -> Integrity`
Creates an `Integrity` object from either string or `Buffer` data, calculating
all the requested hashes and adding any specified options to the object.
`opts.algorithms` determines which algorithms to generate hashes for. All
results will be included in a single `Integrity` object. The default value for
`opts.algorithms` is `['sha512']`. All algorithm strings must be hashes listed
in `crypto.getHashes()` for the host Node.js platform.
`opts.options` may optionally be passed in: it must be an array of option
strings that will be added to all generated integrity hashes generated by
`fromData`. This is a loosely-specified feature of SRIs, and currently has no
specified semantics besides being `?`-separated. Use at your own risk, and
probably avoid if your integrity strings are meant to be used with browsers.
If `opts.strict` is true, the integrity object will be created using strict
parsing rules. See [`ssri.parse`](#parse).
##### Example
```javascript
const integrityObj = ssri.fromData('foobarbaz', {
algorithms: ['sha256', 'sha384', 'sha512']
})
integrity.toString('\n')
// ->
// sha256-l981iLWj8kurw4UbNy8Lpxqdzd7UOxS50Glhv8FwfZ0=
// sha384-irnCxQ0CfQhYGlVAUdwTPC9bF3+YWLxlaDGM4xbYminxpbXEq+D+2GCEBTxcjES9
// sha512-yzd8ELD1piyANiWnmdnpCL5F52f10UfUdEkHywVZeqTt0ymgrxR63Qz0GB7TKPoeeZQmWCaz7T1+9vBnypkYWg==
```
#### <a name="from-stream"></a> `> ssri.fromStream(stream, [opts]) -> Promise<Integrity>`
Returns a Promise of an Integrity object calculated by reading data from
a given `stream`.
It accepts both `opts.algorithms` and `opts.options`, which are documented as
part of [`ssri.fromData`](#from-data).
Additionally, `opts.Promise` may be passed in to inject a Promise library of
choice. By default, ssri will use Node's built-in Promises.
If `opts.strict` is true, the integrity object will be created using strict
parsing rules. See [`ssri.parse`](#parse).
##### Example
```javascript
ssri.fromStream(fs.createReadStream('index.js'), {
algorithms: ['sha1', 'sha512']
}).then(integrity => {
return ssri.checkStream(fs.createReadStream('index.js'), integrity)
}) // succeeds
```
#### <a name="create"></a> `> ssri.create([opts]) -> <Hash>`
Returns a Hash object with `update(<Buffer or string>[,enc])` and `digest()` methods.
The Hash object provides the same methods as [crypto class Hash](https://nodejs.org/dist/latest-v6.x/docs/api/crypto.html#crypto_class_hash).
`digest()` accepts no arguments and returns an Integrity object calculated by reading data from
calls to update.
It accepts both `opts.algorithms` and `opts.options`, which are documented as
part of [`ssri.fromData`](#from-data).
If `opts.strict` is true, the integrity object will be created using strict
parsing rules. See [`ssri.parse`](#parse).
##### Example
```javascript
const integrity = ssri.create().update('foobarbaz').digest()
integrity.toString()
// ->
// sha512-yzd8ELD1piyANiWnmdnpCL5F52f10UfUdEkHywVZeqTt0ymgrxR63Qz0GB7TKPoeeZQmWCaz7T1+9vBnypkYWg==
```
#### <a name="check-data"></a> `> ssri.checkData(data, sri, [opts]) -> Hash|false`
Verifies `data` integrity against an `sri` argument. `data` may be either a
`String` or a `Buffer`, and `sri` can be any subresource integrity
representation that [`ssri.parse`](#parse) can handle.
If verification succeeds, `checkData` will return the name of the algorithm that
was used for verification (a truthy value). Otherwise, it will return `false`.
If `opts.pickAlgorithm` is provided, it will be used by
[`Integrity#pickAlgorithm`](#integrity-pick-algorithm) when deciding which of
the available digests to match against.
If `opts.error` is true, and verification fails, `checkData` will throw either
an `EBADSIZE` or an `EINTEGRITY` error, instead of just returning false.
##### Example
```javascript
const data = fs.readFileSync('index.js')
ssri.checkData(data, ssri.fromData(data)) // -> 'sha512'
ssri.checkData(data, 'sha256-l981iLWj8kurw4UbNy8Lpxqdzd7UOxS50Glhv8FwfZ0')
ssri.checkData(data, 'sha1-BaDDigEST') // -> false
ssri.checkData(data, 'sha1-BaDDigEST', {error: true}) // -> Error! EINTEGRITY
```
#### <a name="check-stream"></a> `> ssri.checkStream(stream, sri, [opts]) -> Promise<Hash>`
Verifies the contents of `stream` against an `sri` argument. `stream` will be
consumed in its entirety by this process. `sri` can be any subresource integrity
representation that [`ssri.parse`](#parse) can handle.
`checkStream` will return a Promise that either resolves to the
`Hash` that succeeded verification, or, if the verification fails
or an error happens with `stream`, the Promise will be rejected.
If the Promise is rejected because verification failed, the returned error will
have `err.code` as `EINTEGRITY`.
If `opts.size` is given, it will be matched against the stream size. An error
with `err.code` `EBADSIZE` will be returned by a rejection if the expected size
and actual size fail to match.
If `opts.pickAlgorithm` is provided, it will be used by
[`Integrity#pickAlgorithm`](#integrity-pick-algorithm) when deciding which of
the available digests to match against.
##### Example
```javascript
const integrity = ssri.fromData(fs.readFileSync('index.js'))
ssri.checkStream(
fs.createReadStream('index.js'),
integrity
)
// ->
// Promise<{
// algorithm: 'sha512',
// digest: 'sha512-yzd8ELD1piyANiWnmdnpCL5F52f10UfUdEkHywVZeqTt0ymgrxR63Qz0GB7TKPoeeZQmWCaz7T1'
// }>
ssri.checkStream(
fs.createReadStream('index.js'),
'sha256-l981iLWj8kurw4UbNy8Lpxqdzd7UOxS50Glhv8FwfZ0'
) // -> Promise<Hash>
ssri.checkStream(
fs.createReadStream('index.js'),
'sha1-BaDDigEST'
) // -> Promise<Error<{code: 'EINTEGRITY'}>>
```
#### <a name="integrity-stream"></a> `> integrityStream([opts]) -> IntegrityStream`
Returns a `Transform` stream that data can be piped through in order to generate
and optionally check data integrity for piped data. When the stream completes
successfully, it emits `size` and `integrity` events, containing the total
number of bytes processed and a calculated `Integrity` instance based on stream
data, respectively.
If `opts.algorithms` is passed in, the listed algorithms will be calculated when
generating the final `Integrity` instance. The default is `['sha512']`.
If `opts.single` is passed in, a single `Hash` instance will be returned.
If `opts.integrity` is passed in, it should be an `integrity` value understood
by [`parse`](#parse) that the stream will check the data against. If
verification succeeds, the integrity stream will emit a `verified` event whose
value is a single `Hash` object that is the one that succeeded verification. If
verification fails, the stream will error with an `EINTEGRITY` error code.
If `opts.size` is given, it will be matched against the stream size. An error
with `err.code` `EBADSIZE` will be emitted by the stream if the expected size
and actual size fail to match.
If `opts.pickAlgorithm` is provided, it will be passed two algorithms as
arguments. ssri will prioritize whichever of the two algorithms is returned by
this function. Note that the function may be called multiple times, and it
**must** return one of the two algorithms provided. By default, ssri will make
a best-effort to pick the strongest/most reliable of the given algorithms. It
may intentionally deprioritize algorithms with known vulnerabilities.
##### Example
```javascript
const integrity = ssri.fromData(fs.readFileSync('index.js'))
fs.createReadStream('index.js')
.pipe(ssri.integrityStream({integrity}))
```
+524
View File
@@ -0,0 +1,524 @@
'use strict'
const crypto = require('crypto')
const MiniPass = require('minipass')
const SPEC_ALGORITHMS = ['sha256', 'sha384', 'sha512']
// TODO: this should really be a hardcoded list of algorithms we support,
// rather than [a-z0-9].
const BASE64_REGEX = /^[a-z0-9+/]+(?:=?=?)$/i
const SRI_REGEX = /^([a-z0-9]+)-([^?]+)([?\S*]*)$/
const STRICT_SRI_REGEX = /^([a-z0-9]+)-([A-Za-z0-9+/=]{44,88})(\?[\x21-\x7E]*)?$/
const VCHAR_REGEX = /^[\x21-\x7E]+$/
const defaultOpts = {
algorithms: ['sha512'],
error: false,
options: [],
pickAlgorithm: getPrioritizedHash,
sep: ' ',
single: false,
strict: false,
}
const ssriOpts = (opts = {}) => ({ ...defaultOpts, ...opts })
const getOptString = options => !options || !options.length
? ''
: `?${options.join('?')}`
const _onEnd = Symbol('_onEnd')
const _getOptions = Symbol('_getOptions')
const _emittedSize = Symbol('_emittedSize')
const _emittedIntegrity = Symbol('_emittedIntegrity')
const _emittedVerified = Symbol('_emittedVerified')
class IntegrityStream extends MiniPass {
constructor (opts) {
super()
this.size = 0
this.opts = opts
// may be overridden later, but set now for class consistency
this[_getOptions]()
// options used for calculating stream. can't be changed.
const { algorithms = defaultOpts.algorithms } = opts
this.algorithms = Array.from(
new Set(algorithms.concat(this.algorithm ? [this.algorithm] : []))
)
this.hashes = this.algorithms.map(crypto.createHash)
}
[_getOptions] () {
const {
integrity,
size,
options,
} = { ...defaultOpts, ...this.opts }
// For verification
this.sri = integrity ? parse(integrity, this.opts) : null
this.expectedSize = size
this.goodSri = this.sri ? !!Object.keys(this.sri).length : false
this.algorithm = this.goodSri ? this.sri.pickAlgorithm(this.opts) : null
this.digests = this.goodSri ? this.sri[this.algorithm] : null
this.optString = getOptString(options)
}
on (ev, handler) {
if (ev === 'size' && this[_emittedSize]) {
return handler(this[_emittedSize])
}
if (ev === 'integrity' && this[_emittedIntegrity]) {
return handler(this[_emittedIntegrity])
}
if (ev === 'verified' && this[_emittedVerified]) {
return handler(this[_emittedVerified])
}
return super.on(ev, handler)
}
emit (ev, data) {
if (ev === 'end') {
this[_onEnd]()
}
return super.emit(ev, data)
}
write (data) {
this.size += data.length
this.hashes.forEach(h => h.update(data))
return super.write(data)
}
[_onEnd] () {
if (!this.goodSri) {
this[_getOptions]()
}
const newSri = parse(this.hashes.map((h, i) => {
return `${this.algorithms[i]}-${h.digest('base64')}${this.optString}`
}).join(' '), this.opts)
// Integrity verification mode
const match = this.goodSri && newSri.match(this.sri, this.opts)
if (typeof this.expectedSize === 'number' && this.size !== this.expectedSize) {
/* eslint-disable-next-line max-len */
const err = new Error(`stream size mismatch when checking ${this.sri}.\n Wanted: ${this.expectedSize}\n Found: ${this.size}`)
err.code = 'EBADSIZE'
err.found = this.size
err.expected = this.expectedSize
err.sri = this.sri
this.emit('error', err)
} else if (this.sri && !match) {
/* eslint-disable-next-line max-len */
const err = new Error(`${this.sri} integrity checksum failed when using ${this.algorithm}: wanted ${this.digests} but got ${newSri}. (${this.size} bytes)`)
err.code = 'EINTEGRITY'
err.found = newSri
err.expected = this.digests
err.algorithm = this.algorithm
err.sri = this.sri
this.emit('error', err)
} else {
this[_emittedSize] = this.size
this.emit('size', this.size)
this[_emittedIntegrity] = newSri
this.emit('integrity', newSri)
if (match) {
this[_emittedVerified] = match
this.emit('verified', match)
}
}
}
}
class Hash {
get isHash () {
return true
}
constructor (hash, opts) {
opts = ssriOpts(opts)
const strict = !!opts.strict
this.source = hash.trim()
// set default values so that we make V8 happy to
// always see a familiar object template.
this.digest = ''
this.algorithm = ''
this.options = []
// 3.1. Integrity metadata (called "Hash" by ssri)
// https://w3c.github.io/webappsec-subresource-integrity/#integrity-metadata-description
const match = this.source.match(
strict
? STRICT_SRI_REGEX
: SRI_REGEX
)
if (!match) {
return
}
if (strict && !SPEC_ALGORITHMS.some(a => a === match[1])) {
return
}
this.algorithm = match[1]
this.digest = match[2]
const rawOpts = match[3]
if (rawOpts) {
this.options = rawOpts.slice(1).split('?')
}
}
hexDigest () {
return this.digest && Buffer.from(this.digest, 'base64').toString('hex')
}
toJSON () {
return this.toString()
}
toString (opts) {
opts = ssriOpts(opts)
if (opts.strict) {
// Strict mode enforces the standard as close to the foot of the
// letter as it can.
if (!(
// The spec has very restricted productions for algorithms.
// https://www.w3.org/TR/CSP2/#source-list-syntax
SPEC_ALGORITHMS.some(x => x === this.algorithm) &&
// Usually, if someone insists on using a "different" base64, we
// leave it as-is, since there's multiple standards, and the
// specified is not a URL-safe variant.
// https://www.w3.org/TR/CSP2/#base64_value
this.digest.match(BASE64_REGEX) &&
// Option syntax is strictly visual chars.
// https://w3c.github.io/webappsec-subresource-integrity/#grammardef-option-expression
// https://tools.ietf.org/html/rfc5234#appendix-B.1
this.options.every(opt => opt.match(VCHAR_REGEX))
)) {
return ''
}
}
const options = this.options && this.options.length
? `?${this.options.join('?')}`
: ''
return `${this.algorithm}-${this.digest}${options}`
}
}
class Integrity {
get isIntegrity () {
return true
}
toJSON () {
return this.toString()
}
isEmpty () {
return Object.keys(this).length === 0
}
toString (opts) {
opts = ssriOpts(opts)
let sep = opts.sep || ' '
if (opts.strict) {
// Entries must be separated by whitespace, according to spec.
sep = sep.replace(/\S+/g, ' ')
}
return Object.keys(this).map(k => {
return this[k].map(hash => {
return Hash.prototype.toString.call(hash, opts)
}).filter(x => x.length).join(sep)
}).filter(x => x.length).join(sep)
}
concat (integrity, opts) {
opts = ssriOpts(opts)
const other = typeof integrity === 'string'
? integrity
: stringify(integrity, opts)
return parse(`${this.toString(opts)} ${other}`, opts)
}
hexDigest () {
return parse(this, { single: true }).hexDigest()
}
// add additional hashes to an integrity value, but prevent
// *changing* an existing integrity hash.
merge (integrity, opts) {
opts = ssriOpts(opts)
const other = parse(integrity, opts)
for (const algo in other) {
if (this[algo]) {
if (!this[algo].find(hash =>
other[algo].find(otherhash =>
hash.digest === otherhash.digest))) {
throw new Error('hashes do not match, cannot update integrity')
}
} else {
this[algo] = other[algo]
}
}
}
match (integrity, opts) {
opts = ssriOpts(opts)
const other = parse(integrity, opts)
const algo = other.pickAlgorithm(opts)
return (
this[algo] &&
other[algo] &&
this[algo].find(hash =>
other[algo].find(otherhash =>
hash.digest === otherhash.digest
)
)
) || false
}
pickAlgorithm (opts) {
opts = ssriOpts(opts)
const pickAlgorithm = opts.pickAlgorithm
const keys = Object.keys(this)
return keys.reduce((acc, algo) => {
return pickAlgorithm(acc, algo) || acc
})
}
}
module.exports.parse = parse
function parse (sri, opts) {
if (!sri) {
return null
}
opts = ssriOpts(opts)
if (typeof sri === 'string') {
return _parse(sri, opts)
} else if (sri.algorithm && sri.digest) {
const fullSri = new Integrity()
fullSri[sri.algorithm] = [sri]
return _parse(stringify(fullSri, opts), opts)
} else {
return _parse(stringify(sri, opts), opts)
}
}
function _parse (integrity, opts) {
// 3.4.3. Parse metadata
// https://w3c.github.io/webappsec-subresource-integrity/#parse-metadata
if (opts.single) {
return new Hash(integrity, opts)
}
const hashes = integrity.trim().split(/\s+/).reduce((acc, string) => {
const hash = new Hash(string, opts)
if (hash.algorithm && hash.digest) {
const algo = hash.algorithm
if (!acc[algo]) {
acc[algo] = []
}
acc[algo].push(hash)
}
return acc
}, new Integrity())
return hashes.isEmpty() ? null : hashes
}
module.exports.stringify = stringify
function stringify (obj, opts) {
opts = ssriOpts(opts)
if (obj.algorithm && obj.digest) {
return Hash.prototype.toString.call(obj, opts)
} else if (typeof obj === 'string') {
return stringify(parse(obj, opts), opts)
} else {
return Integrity.prototype.toString.call(obj, opts)
}
}
module.exports.fromHex = fromHex
function fromHex (hexDigest, algorithm, opts) {
opts = ssriOpts(opts)
const optString = getOptString(opts.options)
return parse(
`${algorithm}-${
Buffer.from(hexDigest, 'hex').toString('base64')
}${optString}`, opts
)
}
module.exports.fromData = fromData
function fromData (data, opts) {
opts = ssriOpts(opts)
const algorithms = opts.algorithms
const optString = getOptString(opts.options)
return algorithms.reduce((acc, algo) => {
const digest = crypto.createHash(algo).update(data).digest('base64')
const hash = new Hash(
`${algo}-${digest}${optString}`,
opts
)
/* istanbul ignore else - it would be VERY strange if the string we
* just calculated with an algo did not have an algo or digest.
*/
if (hash.algorithm && hash.digest) {
const hashAlgo = hash.algorithm
if (!acc[hashAlgo]) {
acc[hashAlgo] = []
}
acc[hashAlgo].push(hash)
}
return acc
}, new Integrity())
}
module.exports.fromStream = fromStream
function fromStream (stream, opts) {
opts = ssriOpts(opts)
const istream = integrityStream(opts)
return new Promise((resolve, reject) => {
stream.pipe(istream)
stream.on('error', reject)
istream.on('error', reject)
let sri
istream.on('integrity', s => {
sri = s
})
istream.on('end', () => resolve(sri))
istream.on('data', () => {})
})
}
module.exports.checkData = checkData
function checkData (data, sri, opts) {
opts = ssriOpts(opts)
sri = parse(sri, opts)
if (!sri || !Object.keys(sri).length) {
if (opts.error) {
throw Object.assign(
new Error('No valid integrity hashes to check against'), {
code: 'EINTEGRITY',
}
)
} else {
return false
}
}
const algorithm = sri.pickAlgorithm(opts)
const digest = crypto.createHash(algorithm).update(data).digest('base64')
const newSri = parse({ algorithm, digest })
const match = newSri.match(sri, opts)
if (match || !opts.error) {
return match
} else if (typeof opts.size === 'number' && (data.length !== opts.size)) {
/* eslint-disable-next-line max-len */
const err = new Error(`data size mismatch when checking ${sri}.\n Wanted: ${opts.size}\n Found: ${data.length}`)
err.code = 'EBADSIZE'
err.found = data.length
err.expected = opts.size
err.sri = sri
throw err
} else {
/* eslint-disable-next-line max-len */
const err = new Error(`Integrity checksum failed when using ${algorithm}: Wanted ${sri}, but got ${newSri}. (${data.length} bytes)`)
err.code = 'EINTEGRITY'
err.found = newSri
err.expected = sri
err.algorithm = algorithm
err.sri = sri
throw err
}
}
module.exports.checkStream = checkStream
function checkStream (stream, sri, opts) {
opts = ssriOpts(opts)
opts.integrity = sri
sri = parse(sri, opts)
if (!sri || !Object.keys(sri).length) {
return Promise.reject(Object.assign(
new Error('No valid integrity hashes to check against'), {
code: 'EINTEGRITY',
}
))
}
const checker = integrityStream(opts)
return new Promise((resolve, reject) => {
stream.pipe(checker)
stream.on('error', reject)
checker.on('error', reject)
let verified
checker.on('verified', s => {
verified = s
})
checker.on('end', () => resolve(verified))
checker.on('data', () => {})
})
}
module.exports.integrityStream = integrityStream
function integrityStream (opts = {}) {
return new IntegrityStream(opts)
}
module.exports.create = createIntegrity
function createIntegrity (opts) {
opts = ssriOpts(opts)
const algorithms = opts.algorithms
const optString = getOptString(opts.options)
const hashes = algorithms.map(crypto.createHash)
return {
update: function (chunk, enc) {
hashes.forEach(h => h.update(chunk, enc))
return this
},
digest: function (enc) {
const integrity = algorithms.reduce((acc, algo) => {
const digest = hashes.shift().digest('base64')
const hash = new Hash(
`${algo}-${digest}${optString}`,
opts
)
/* istanbul ignore else - it would be VERY strange if the hash we
* just calculated with an algo did not have an algo or digest.
*/
if (hash.algorithm && hash.digest) {
const hashAlgo = hash.algorithm
if (!acc[hashAlgo]) {
acc[hashAlgo] = []
}
acc[hashAlgo].push(hash)
}
return acc
}, new Integrity())
return integrity
},
}
}
const NODE_HASHES = new Set(crypto.getHashes())
// This is a Best Effort™ at a reasonable priority for hash algos
const DEFAULT_PRIORITY = [
'md5', 'whirlpool', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512',
// TODO - it's unclear _which_ of these Node will actually use as its name
// for the algorithm, so we guesswork it based on the OpenSSL names.
'sha3',
'sha3-256', 'sha3-384', 'sha3-512',
'sha3_256', 'sha3_384', 'sha3_512',
].filter(algo => NODE_HASHES.has(algo))
function getPrioritizedHash (algo1, algo2) {
/* eslint-disable-next-line max-len */
return DEFAULT_PRIORITY.indexOf(algo1.toLowerCase()) >= DEFAULT_PRIORITY.indexOf(algo2.toLowerCase())
? algo1
: algo2
}
@@ -0,0 +1,63 @@
{
"name": "ssri",
"version": "9.0.1",
"description": "Standard Subresource Integrity library -- parses, serializes, generates, and verifies integrity metadata according to the SRI spec.",
"main": "lib/index.js",
"files": [
"bin/",
"lib/"
],
"scripts": {
"prerelease": "npm t",
"postrelease": "npm publish",
"prepublishOnly": "git push origin --follow-tags",
"posttest": "npm run lint",
"test": "tap",
"coverage": "tap",
"lint": "eslint \"**/*.js\"",
"postlint": "template-oss-check",
"template-oss-apply": "template-oss-apply --force",
"lintfix": "npm run lint -- --fix",
"preversion": "npm test",
"postversion": "npm publish",
"snap": "tap"
},
"tap": {
"check-coverage": true
},
"repository": {
"type": "git",
"url": "https://github.com/npm/ssri.git"
},
"keywords": [
"w3c",
"web",
"security",
"integrity",
"checksum",
"hashing",
"subresource integrity",
"sri",
"sri hash",
"sri string",
"sri generator",
"html"
],
"author": "GitHub Inc.",
"license": "ISC",
"dependencies": {
"minipass": "^3.1.1"
},
"devDependencies": {
"@npmcli/eslint-config": "^3.0.1",
"@npmcli/template-oss": "3.5.0",
"tap": "^16.0.1"
},
"engines": {
"node": "^12.13.0 || ^14.15.0 || >=16.0.0"
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "3.5.0"
}
}
@@ -0,0 +1,5 @@
Copyright npm, Inc
Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
@@ -0,0 +1,34 @@
unique-filename
===============
Generate a unique filename for use in temporary directories or caches.
```js
const uniqueFilename = require('unique-filename')
// returns something like: '/tmp/c5b28f47'
const randomTmpfile = uniqueFilename(os.tmpdir())
// returns something like: '/tmp/my-test-51a7b48d'
const randomPrefixedTmpfile = uniqueFilename(os.tmpdir(), 'my-test')
// returns something like: '/my-tmp-dir/testing-7ddd44c0'
const uniqueTmpfile = uniqueFilename('/my-tmp-dir', 'testing', '/my/thing/to/uniq/on')
```
### uniqueFilename(*dir*, *fileprefix*, *uniqstr*) → String
Returns the full path of a unique filename that looks like:
`dir/prefix-7ddd44c0`
or `dir/7ddd44c0`
*dir* – The path you want the filename in. `os.tmpdir()` is a good choice for this.
*fileprefix* – A string to append prior to the unique part of the filename.
The parameter is required if *uniqstr* is also passed in but is otherwise
optional and can be `undefined`/`null`/`''`. If present and not empty
then this string plus a hyphen are prepended to the unique part.
*uniqstr* – Optional, if not passed the unique part of the resulting
filename will be random. If passed in it will be generated from this string
in a reproducible way.
@@ -0,0 +1,7 @@
var path = require('path')
var uniqueSlug = require('unique-slug')
module.exports = function (filepath, prefix, uniq) {
return path.join(filepath, (prefix ? prefix + '-' : '') + uniqueSlug(uniq))
}
@@ -0,0 +1,48 @@
{
"name": "unique-filename",
"version": "2.0.1",
"description": "Generate a unique filename for use in temporary directories or caches.",
"main": "lib/index.js",
"scripts": {
"test": "tap",
"lint": "eslint \"**/*.js\"",
"postlint": "template-oss-check",
"template-oss-apply": "template-oss-apply --force",
"lintfix": "npm run lint -- --fix",
"preversion": "npm test",
"postversion": "npm publish",
"prepublishOnly": "git push origin --follow-tags",
"snap": "tap",
"posttest": "npm run lint"
},
"repository": {
"type": "git",
"url": "https://github.com/npm/unique-filename.git"
},
"keywords": [],
"author": "GitHub Inc.",
"license": "ISC",
"bugs": {
"url": "https://github.com/iarna/unique-filename/issues"
},
"homepage": "https://github.com/iarna/unique-filename",
"devDependencies": {
"@npmcli/eslint-config": "^3.1.0",
"@npmcli/template-oss": "3.5.0",
"tap": "^16.3.0"
},
"dependencies": {
"unique-slug": "^3.0.0"
},
"files": [
"bin/",
"lib/"
],
"engines": {
"node": "^12.13.0 || ^14.15.0 || >=16.0.0"
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "3.5.0"
}
}
@@ -0,0 +1,15 @@
The ISC License
Copyright npm, Inc
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
@@ -0,0 +1,19 @@
unique-slug
===========
Generate a unique character string suitible for use in files and URLs.
```
var uniqueSlug = require('unique-slug')
var randomSlug = uniqueSlug()
var fileSlug = uniqueSlug('/etc/passwd')
```
### uniqueSlug(*str*) → String (8 chars)
If *str* is passed in then the return value will be its murmur hash in
hex.
If *str* is not passed in, it will be 4 randomly generated bytes
converted into 8 hexadecimal characters.
@@ -0,0 +1,11 @@
'use strict'
var MurmurHash3 = require('imurmurhash')
module.exports = function (uniq) {
if (uniq) {
var hash = new MurmurHash3(uniq)
return ('00000000' + hash.result().toString(16)).slice(-8)
} else {
return (Math.random().toString(16) + '0000000').slice(2, 10)
}
}
@@ -0,0 +1,44 @@
{
"name": "unique-slug",
"version": "3.0.0",
"description": "Generate a unique character string suitible for use in files and URLs.",
"main": "lib/index.js",
"scripts": {
"test": "tap",
"lint": "eslint \"**/*.js\"",
"postlint": "template-oss-check",
"template-oss-apply": "template-oss-apply --force",
"lintfix": "npm run lint -- --fix",
"preversion": "npm test",
"postversion": "npm publish",
"prepublishOnly": "git push origin --follow-tags",
"snap": "tap",
"posttest": "npm run lint"
},
"keywords": [],
"author": "GitHub Inc.",
"license": "ISC",
"devDependencies": {
"@npmcli/eslint-config": "^3.1.0",
"@npmcli/template-oss": "3.5.0",
"tap": "^16.3.0"
},
"repository": {
"type": "git",
"url": "https://github.com/npm/unique-slug.git"
},
"dependencies": {
"imurmurhash": "^0.1.4"
},
"files": [
"bin/",
"lib/"
],
"engines": {
"node": "^12.13.0 || ^14.15.0 || >=16.0.0"
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "3.5.0"
}
}
+79
View File
@@ -0,0 +1,79 @@
{
"name": "make-fetch-happen",
"version": "10.2.1",
"description": "Opinionated, caching, retrying fetch client",
"main": "lib/index.js",
"files": [
"bin/",
"lib/"
],
"scripts": {
"preversion": "npm test",
"postversion": "npm publish",
"prepublishOnly": "git push origin --follow-tags",
"test": "tap",
"posttest": "npm run lint",
"eslint": "eslint",
"lint": "eslint \"**/*.js\"",
"lintfix": "npm run lint -- --fix",
"postlint": "template-oss-check",
"snap": "tap",
"template-oss-apply": "template-oss-apply --force"
},
"repository": {
"type": "git",
"url": "https://github.com/npm/make-fetch-happen.git"
},
"keywords": [
"http",
"request",
"fetch",
"mean girls",
"caching",
"cache",
"subresource integrity"
],
"author": "GitHub Inc.",
"license": "ISC",
"dependencies": {
"agentkeepalive": "^4.2.1",
"cacache": "^16.1.0",
"http-cache-semantics": "^4.1.0",
"http-proxy-agent": "^5.0.0",
"https-proxy-agent": "^5.0.0",
"is-lambda": "^1.0.1",
"lru-cache": "^7.7.1",
"minipass": "^3.1.6",
"minipass-collect": "^1.0.2",
"minipass-fetch": "^2.0.3",
"minipass-flush": "^1.0.5",
"minipass-pipeline": "^1.2.4",
"negotiator": "^0.6.3",
"promise-retry": "^2.0.1",
"socks-proxy-agent": "^7.0.0",
"ssri": "^9.0.0"
},
"devDependencies": {
"@npmcli/eslint-config": "^3.0.1",
"@npmcli/template-oss": "3.5.0",
"mkdirp": "^1.0.4",
"nock": "^13.2.4",
"rimraf": "^3.0.2",
"safe-buffer": "^5.2.1",
"standard-version": "^9.3.2",
"tap": "^16.0.0"
},
"engines": {
"node": "^12.13.0 || ^14.15.0 || >=16.0.0"
},
"tap": {
"color": 1,
"files": "test/*.js",
"check-coverage": true,
"timeout": 60
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "3.5.0"
}
}