Changing front

This commit is contained in:
ewandor committed 2023-01-16 17:49:38 +01:00
1 parent 0b8a93b256
commit 4fe4be7730
48586 files changed
+4725790 -17464

No files matched your search

+20
View File
@@ -0,0 +1,20 @@
<!-- This file is automatically added by @npmcli/template-oss. Do not edit. -->
ISC License
Copyright npm, Inc.
Permission to use, copy, modify, and/or distribute this
software for any purpose with or without fee is hereby
granted, provided that the above copyright notice and this
permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND NPM DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO
EVENT SHALL NPM BE LIABLE FOR ANY SPECIAL, DIRECT,
INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE
USE OR PERFORMANCE OF THIS SOFTWARE.
+615
View File
@@ -0,0 +1,615 @@
# npm-registry-fetch
[`npm-registry-fetch`](https://github.com/npm/npm-registry-fetch) is a Node.js
library that implements a `fetch`-like API for accessing npm registry APIs
consistently. It's able to consume npm-style configuration values and has all
the necessary logic for picking registries, handling scopes, and dealing with
authentication details built-in.
This package is meant to replace the older
[`npm-registry-client`](https://npm.im/npm-registry-client).
## Example
```javascript
const npmFetch = require('npm-registry-fetch')
console.log(
await npmFetch.json('/-/ping')
)
```
## Table of Contents
* [Installing](#install)
* [Example](#example)
* [Contributing](#contributing)
* [API](#api)
* [`fetch`](#fetch)
* [`fetch.json`](#fetch-json)
* [`fetch` options](#fetch-opts)
### Install
`$ npm install npm-registry-fetch`
### Contributing
The npm team enthusiastically welcomes contributions and project participation!
There's a bunch of things you can do if you want to contribute! The [Contributor
Guide](CONTRIBUTING.md) has all the information you need for everything from
reporting bugs to contributing entire new features. Please don't hesitate to
jump in if you'd like to, or even ask us questions if something isn't clear.
All participants and maintainers in this project are expected to follow [Code of
Conduct](CODE_OF_CONDUCT.md), and just generally be excellent to each other.
Please refer to the [Changelog](CHANGELOG.md) for project history details, too.
Happy hacking!
### API
#### Caching and `write=true` query strings
Before performing any PUT or DELETE operation, npm clients first make a
GET request to the registry resource being updated, which includes
the query string `?write=true`.
The semantics of this are, effectively, "I intend to write to this thing,
and need to know the latest current value, so that my write can land
cleanly".
The public npm registry handles these `?write=true` requests by ensuring
that the cache is re-validated before sending a response. In order to
maintain the same behavior on the client, and not get tripped up by an
overeager local cache when we intend to write data to the registry, any
request that comes through `npm-registry-fetch` that contains `write=true`
in the query string will forcibly set the `prefer-online` option to `true`,
and set both `prefer-offline` and `offline` to false, so that any local
cached value will be revalidated.
#### <a name="fetch"></a> `> fetch(url, [opts]) -> Promise<Response>`
Performs a request to a given URL.
The URL can be either a full URL, or a path to one. The appropriate registry
will be automatically picked if only a URL path is given.
For available options, please see the section on [`fetch` options](#fetch-opts).
##### Example
```javascript
const res = await fetch('/-/ping')
console.log(res.headers)
res.on('data', d => console.log(d.toString('utf8')))
```
#### <a name="fetch-json"></a> `> fetch.json(url, [opts]) -> Promise<ResponseJSON>`
Performs a request to a given registry URL, parses the body of the response as
JSON, and returns it as its final value. This is a utility shorthand for
`fetch(url).then(res => res.json())`.
For available options, please see the section on [`fetch` options](#fetch-opts).
##### Example
```javascript
const res = await fetch.json('/-/ping')
console.log(res) // Body parsed as JSON
```
#### <a name="fetch-json-stream"></a> `> fetch.json.stream(url, jsonPath, [opts]) -> Stream`
Performs a request to a given registry URL and parses the body of the response
as JSON, with each entry being emitted through the stream.
The `jsonPath` argument is a [`JSONStream.parse()`
path](https://github.com/dominictarr/JSONStream#jsonstreamparsepath), and the
returned stream (unlike default `JSONStream`s), has a valid
`Symbol.asyncIterator` implementation.
For available options, please see the section on [`fetch` options](#fetch-opts).
##### Example
```javascript
console.log('https://npm.im/~zkat has access to the following packages:')
for await (let {key, value} of fetch.json.stream('/-/user/zkat/package', '$*')) {
console.log(`https://npm.im/${key} (perms: ${value})`)
}
```
#### <a name="fetch-opts"></a> `fetch` Options
Fetch options are optional, and can be passed in as either a Map-like object
(one with a `.get()` method), a plain javascript object, or a
[`figgy-pudding`](https://npm.im/figgy-pudding) instance.
##### <a name="opts-agent"></a> `opts.agent`
* Type: http.Agent
* Default: an appropriate agent based on URL protocol and proxy settings
An [`Agent`](https://nodejs.org/api/http.html#http_class_http_agent) instance to
be shared across requests. This allows multiple concurrent `fetch` requests to
happen on the same socket.
You do _not_ need to provide this option unless you want something particularly
specialized, since proxy configurations and http/https agents are already
automatically managed internally when this option is not passed through.
##### <a name="opts-body"></a> `opts.body`
* Type: Buffer | Stream | Object
* Default: null
Request body to send through the outgoing request. Buffers and Streams will be
passed through as-is, with a default `content-type` of
`application/octet-stream`. Plain JavaScript objects will be `JSON.stringify`ed
and the `content-type` will default to `application/json`.
Use [`opts.headers`](#opts-headers) to set the content-type to something else.
##### <a name="opts-ca"></a> `opts.ca`
* Type: String, Array, or null
* Default: null
The Certificate Authority signing certificate that is trusted for SSL
connections to the registry. Values should be in PEM format (Windows calls it
"Base-64 encoded X.509 (.CER)") with newlines replaced by the string `'\n'`. For
example:
```
{
ca: '-----BEGIN CERTIFICATE-----\nXXXX\nXXXX\n-----END CERTIFICATE-----'
}
```
Set to `null` to only allow "known" registrars, or to a specific CA cert
to trust only that specific signing authority.
Multiple CAs can be trusted by specifying an array of certificates instead of a
single string.
See also [`opts.strictSSL`](#opts-strictSSL), [`opts.ca`](#opts-ca) and
[`opts.key`](#opts-key)
##### <a name="opts-cache"></a> `opts.cache`
* Type: path
* Default: null
The location of the http cache directory. If provided, certain cachable requests
will be cached according to [IETF RFC 7234](https://tools.ietf.org/html/rfc7234)
rules. This will speed up future requests, as well as make the cached data
available offline if necessary/requested.
See also [`offline`](#opts-offline), [`preferOffline`](#opts-preferOffline),
and [`preferOnline`](#opts-preferOnline).
##### <a name="opts-cert"></a> `opts.cert`
* Type: String
* Default: null
A client certificate to pass when accessing the registry. Values should be in
PEM format (Windows calls it "Base-64 encoded X.509 (.CER)") with newlines
replaced by the string `'\n'`. For example:
```
{
cert: '-----BEGIN CERTIFICATE-----\nXXXX\nXXXX\n-----END CERTIFICATE-----'
}
```
It is _not_ the path to a certificate file (and there is no "certfile" option).
See also: [`opts.ca`](#opts-ca) and [`opts.key`](#opts-key)
##### <a name="opts-fetchRetries"></a> `opts.fetchRetries`
* Type: Number
* Default: 2
The "retries" config for [`retry`](https://npm.im/retry) to use when fetching
packages from the registry.
See also [`opts.retry`](#opts-retry) to provide all retry options as a single
object.
##### <a name="opts-fetchRetryFactor"></a> `opts.fetchRetryFactor`
* Type: Number
* Default: 10
The "factor" config for [`retry`](https://npm.im/retry) to use when fetching
packages.
See also [`opts.retry`](#opts-retry) to provide all retry options as a single
object.
##### <a name="opts-fetchRetryMintimeout"></a> `opts.fetchRetryMintimeout`
* Type: Number
* Default: 10000 (10 seconds)
The "minTimeout" config for [`retry`](https://npm.im/retry) to use when fetching
packages.
See also [`opts.retry`](#opts-retry) to provide all retry options as a single
object.
##### <a name="opts-fetchRetryMaxtimeout"></a> `opts.fetchRetryMaxtimeout`
* Type: Number
* Default: 60000 (1 minute)
The "maxTimeout" config for [`retry`](https://npm.im/retry) to use when fetching
packages.
See also [`opts.retry`](#opts-retry) to provide all retry options as a single
object.
##### <a name="opts-forceAuth"></a> `opts.forceAuth`
* Type: Object
* Default: null
If present, other auth-related values in `opts` will be completely ignored,
including `alwaysAuth`, `email`, and `otp`, when calculating auth for a request,
and the auth details in `opts.forceAuth` will be used instead.
##### <a name="opts-gzip"></a> `opts.gzip`
* Type: Boolean
* Default: false
If true, `npm-registry-fetch` will set the `Content-Encoding` header to `gzip`
and use `zlib.gzip()` or `zlib.createGzip()` to gzip-encode
[`opts.body`](#opts-body).
##### <a name="opts-headers"></a> `opts.headers`
* Type: Object
* Default: null
Additional headers for the outgoing request. This option can also be used to
override headers automatically generated by `npm-registry-fetch`, such as
`Content-Type`.
##### <a name="opts-ignoreBody"></a> `opts.ignoreBody`
* Type: Boolean
* Default: false
If true, the **response body** will be thrown away and `res.body` set to `null`.
This will prevent dangling response sockets for requests where you don't usually
care what the response body is.
##### <a name="opts-integrity"></a> `opts.integrity`
* Type: String | [SRI object](https://npm.im/ssri)
* Default: null
If provided, the response body's will be verified against this integrity string,
using [`ssri`](https://npm.im/ssri). If verification succeeds, the response will
complete as normal. If verification fails, the response body will error with an
`EINTEGRITY` error.
Body integrity is only verified if the body is actually consumed to completion --
that is, if you use `res.json()`/`res.buffer()`, or if you consume the default
`res` stream data to its end.
Cached data will have its integrity automatically verified using the
previously-generated integrity hash for the saved request information, so
`EINTEGRITY` errors can happen if [`opts.cache`](#opts-cache) is used, even if
`opts.integrity` is not passed in.
##### <a name="opts-key"></a> `opts.key`
* Type: String
* Default: null
A client key to pass when accessing the registry. Values should be in PEM
format with newlines replaced by the string `'\n'`. For example:
```
{
key: '-----BEGIN PRIVATE KEY-----\nXXXX\nXXXX\n-----END PRIVATE KEY-----'
}
```
It is _not_ the path to a key file (and there is no "keyfile" option).
See also: [`opts.ca`](#opts-ca) and [`opts.cert`](#opts-cert)
##### <a name="opts-localAddress"></a> `opts.localAddress`
* Type: IP Address String
* Default: null
The IP address of the local interface to use when making connections
to the registry.
See also [`opts.proxy`](#opts-proxy)
##### <a name="opts-mapJSON"></a> `opts.mapJSON`
* Type: Function
* Default: undefined
When using `fetch.json.stream()` (NOT `fetch.json()`), this will be passed down
to [`JSONStream`](https://npm.im/JSONStream) as the second argument to
`JSONStream.parse`, and can be used to transform stream data before output.
##### <a name="opts-maxSockets"></a> `opts.maxSockets`
* Type: Integer
* Default: 12
Maximum number of sockets to keep open during requests. Has no effect if
[`opts.agent`](#opts-agent) is used.
##### <a name="opts-method"></a> `opts.method`
* Type: String
* Default: 'GET'
HTTP method to use for the outgoing request. Case-insensitive.
##### <a name="opts-noproxy"></a> `opts.noproxy`
* Type: Boolean
* Default: process.env.NOPROXY
If true, proxying will be disabled even if [`opts.proxy`](#opts-proxy) is used.
##### <a name="opts-npmSession"></a> `opts.npmSession`
* Type: String
* Default: null
If provided, will be sent in the `npm-session` header. This header is used by
the npm registry to identify individual user sessions (usually individual
invocations of the CLI).
##### <a name="opts-npmCommand"></a> `opts.npmCommand`
* Type: String
* Default: null
If provided, it will be sent in the `npm-command` header. This header is
used by the npm registry to identify the npm command that caused this
request to be made.
##### <a name="opts-offline"></a> `opts.offline`
* Type: Boolean
* Default: false
Force offline mode: no network requests will be done during install. To allow
`npm-registry-fetch` to fill in missing cache data, see
[`opts.preferOffline`](#opts-preferOffline).
This option is only really useful if you're also using
[`opts.cache`](#opts-cache).
This option is set to `true` when the request includes `write=true` in the
query string.
##### <a name="opts-otp"></a> `opts.otp`
* Type: Number | String
* Default: null
This is a one-time password from a two-factor authenticator. It is required for
certain registry interactions when two-factor auth is enabled for a user
account.
##### <a name="opts-otpPrompt"></a> `opts.otpPrompt`
* Type: Function
* Default: null
This is a method which will be called to provide an OTP if the server
responds with a 401 response indicating that a one-time-password is
required.
It may return a promise, which must resolve to the OTP value to be used.
If the method fails to provide an OTP value, then the fetch will fail with
the auth error that indicated an OTP was needed.
##### <a name="opts-password"></a> `opts.password`
* Alias: `_password`
* Type: String
* Default: null
Password used for basic authentication. For the more modern authentication
method, please use the (more secure) [`opts.token`](#opts-token)
Can optionally be scoped to a registry by using a "nerf dart" for that registry.
That is:
```
{
'//registry.npmjs.org/:password': 't0k3nH34r'
}
```
See also [`opts.username`](#opts-username)
##### <a name="opts-preferOffline"></a> `opts.preferOffline`
* Type: Boolean
* Default: false
If true, staleness checks for cached data will be bypassed, but missing data
will be requested from the server. To force full offline mode, use
[`opts.offline`](#opts-offline).
This option is generally only useful if you're also using
[`opts.cache`](#opts-cache).
This option is set to `false` when the request includes `write=true` in the
query string.
##### <a name="opts-preferOnline"></a> `opts.preferOnline`
* Type: Boolean
* Default: false
If true, staleness checks for cached data will be forced, making the CLI look
for updates immediately even for fresh package data.
This option is generally only useful if you're also using
[`opts.cache`](#opts-cache).
This option is set to `true` when the request includes `write=true` in the
query string.
##### <a name="opts-scope"></a> `opts.scope`
* Type: String
* Default: null
If provided, will be sent in the `npm-scope` header. This header is used by the
npm registry to identify the toplevel package scope that a particular project
installation is using.
##### <a name="opts-proxy"></a> `opts.proxy`
* Type: url
* Default: null
A proxy to use for outgoing http requests. If not passed in, the `HTTP(S)_PROXY`
environment variable will be used.
##### <a name="opts-query"></a> `opts.query`
* Type: String | Object
* Default: null
If provided, the request URI will have a query string appended to it using this
query. If `opts.query` is an object, it will be converted to a query string
using
[`querystring.stringify()`](https://nodejs.org/api/querystring.html#querystring_querystring_stringify_obj_sep_eq_options).
If the request URI already has a query string, it will be merged with
`opts.query`, preferring `opts.query` values.
##### <a name="opts-registry"></a> `opts.registry`
* Type: URL
* Default: `'https://registry.npmjs.org'`
Registry configuration for a request. If a request URL only includes the URL
path, this registry setting will be prepended.
See also [`opts.scope`](#opts-scope), [`opts.spec`](#opts-spec), and
[`opts.<scope>:registry`](#opts-scope-registry) which can all affect the actual
registry URL used by the outgoing request.
##### <a name="opts-retry"></a> `opts.retry`
* Type: Object
* Default: null
Single-object configuration for request retry settings. If passed in, will
override individually-passed `fetch-retry-*` settings.
##### <a name="opts-scope"></a> `opts.scope`
* Type: String
* Default: null
Associate an operation with a scope for a scoped registry. This option can force
lookup of scope-specific registries and authentication.
See also [`opts.<scope>:registry`](#opts-scope-registry) and
[`opts.spec`](#opts-spec) for interactions with this option.
##### <a name="opts-scope-registry"></a> `opts.<scope>:registry`
* Type: String
* Default: null
This option type can be used to configure the registry used for requests
involving a particular scope. For example, `opts['@myscope:registry'] =
'https://scope-specific.registry/'` will make it so requests go out to this
registry instead of [`opts.registry`](#opts-registry) when
[`opts.scope`](#opts-scope) is used, or when [`opts.spec`](#opts-spec) is a
scoped package spec.
The `@` before the scope name is optional, but recommended.
##### <a name="opts-spec"></a> `opts.spec`
* Type: String | [`npm-registry-arg`](https://npm.im/npm-registry-arg) object.
* Default: null
If provided, can be used to automatically configure [`opts.scope`](#opts-scope)
based on a specific package name. Non-registry package specs will throw an
error.
##### <a name="opts-strictSSL"></a> `opts.strictSSL`
* Type: Boolean
* Default: true
Whether or not to do SSL key validation when making requests to the
registry via https.
See also [`opts.ca`](#opts-ca).
##### <a name="opts-timeout"></a> `opts.timeout`
* Type: Milliseconds
* Default: 300000 (5 minutes)
Time before a hanging request times out.
##### <a name="opts-authtoken"></a> `opts._authToken`
* Type: String
* Default: null
Authentication token string.
Can be scoped to a registry by using a "nerf dart" for that registry. That is:
```
{
'//registry.npmjs.org/:_authToken': 't0k3nH34r'
}
```
##### <a name="opts-userAgent"></a> `opts.userAgent`
* Type: String
* Default: `'npm-registry-fetch@<version>/node@<node-version>+<arch> (<platform>)'`
User agent string to send in the `User-Agent` header.
##### <a name="opts-username"></a> `opts.username`
* Type: String
* Default: null
Username used for basic authentication. For the more modern authentication
method, please use the (more secure) [`opts.authtoken`](#opts-authtoken)
Can optionally be scoped to a registry by using a "nerf dart" for that registry.
That is:
```
{
'//registry.npmjs.org/:username': 't0k3nH34r'
}
```
See also [`opts.password`](#opts-password)
+145
View File
@@ -0,0 +1,145 @@
'use strict'
const fs = require('fs')
const npa = require('npm-package-arg')
const { URL } = require('url')
// Find the longest registry key that is used for some kind of auth
// in the options.
const regKeyFromURI = (uri, opts) => {
const parsed = new URL(uri)
// try to find a config key indicating we have auth for this registry
// can be one of :_authToken, :_auth, :_password and :username, or
// :certfile and :keyfile
// We walk up the "path" until we're left with just //<host>[:<port>],
// stopping when we reach '//'.
let regKey = `//${parsed.host}${parsed.pathname}`
while (regKey.length > '//'.length) {
// got some auth for this URI
if (hasAuth(regKey, opts)) {
return regKey
}
// can be either //host/some/path/:_auth or //host/some/path:_auth
// walk up by removing EITHER what's after the slash OR the slash itself
regKey = regKey.replace(/([^/]+|\/)$/, '')
}
}
const hasAuth = (regKey, opts) => (
opts[`${regKey}:_authToken`] ||
opts[`${regKey}:_auth`] ||
opts[`${regKey}:username`] && opts[`${regKey}:_password`] ||
opts[`${regKey}:certfile`] && opts[`${regKey}:keyfile`]
)
const sameHost = (a, b) => {
const parsedA = new URL(a)
const parsedB = new URL(b)
return parsedA.host === parsedB.host
}
const getRegistry = opts => {
const { spec } = opts
const { scope: specScope, subSpec } = spec ? npa(spec) : {}
const subSpecScope = subSpec && subSpec.scope
const scope = subSpec ? subSpecScope : specScope
const scopeReg = scope && opts[`${scope}:registry`]
return scopeReg || opts.registry
}
const maybeReadFile = file => {
try {
return fs.readFileSync(file, 'utf8')
} catch (er) {
if (er.code !== 'ENOENT') {
throw er
}
return null
}
}
const getAuth = (uri, opts = {}) => {
const { forceAuth } = opts
if (!uri) {
throw new Error('URI is required')
}
const regKey = regKeyFromURI(uri, forceAuth || opts)
// we are only allowed to use what's in forceAuth if specified
if (forceAuth && !regKey) {
return new Auth({
scopeAuthKey: null,
token: forceAuth._authToken || forceAuth.token,
username: forceAuth.username,
password: forceAuth._password || forceAuth.password,
auth: forceAuth._auth || forceAuth.auth,
certfile: forceAuth.certfile,
keyfile: forceAuth.keyfile,
})
}
// no auth for this URI, but might have it for the registry
if (!regKey) {
const registry = getRegistry(opts)
if (registry && uri !== registry && sameHost(uri, registry)) {
return getAuth(registry, opts)
} else if (registry !== opts.registry) {
// If making a tarball request to a different base URI than the
// registry where we logged in, but the same auth SHOULD be sent
// to that artifact host, then we track where it was coming in from,
// and warn the user if we get a 4xx error on it.
const scopeAuthKey = regKeyFromURI(registry, opts)
return new Auth({ scopeAuthKey })
}
}
const {
[`${regKey}:_authToken`]: token,
[`${regKey}:username`]: username,
[`${regKey}:_password`]: password,
[`${regKey}:_auth`]: auth,
[`${regKey}:certfile`]: certfile,
[`${regKey}:keyfile`]: keyfile,
} = opts
return new Auth({
scopeAuthKey: null,
token,
auth,
username,
password,
certfile,
keyfile,
})
}
class Auth {
constructor ({ token, auth, username, password, scopeAuthKey, certfile, keyfile }) {
this.scopeAuthKey = scopeAuthKey
this.token = null
this.auth = null
this.isBasicAuth = false
this.cert = null
this.key = null
if (token) {
this.token = token
} else if (auth) {
this.auth = auth
} else if (username && password) {
const p = Buffer.from(password, 'base64').toString('utf8')
this.auth = Buffer.from(`${username}:${p}`, 'utf8').toString('base64')
this.isBasicAuth = true
}
// mTLS may be used in conjunction with another auth method above
if (certfile && keyfile) {
const cert = maybeReadFile(certfile, 'utf-8')
const key = maybeReadFile(keyfile, 'utf-8')
if (cert && key) {
this.cert = cert
this.key = key
}
}
}
}
module.exports = getAuth
+100
View File
@@ -0,0 +1,100 @@
'use strict'
const errors = require('./errors.js')
const { Response } = require('minipass-fetch')
const defaultOpts = require('./default-opts.js')
const log = require('proc-log')
const cleanUrl = require('./clean-url.js')
/* eslint-disable-next-line max-len */
const moreInfoUrl = 'https://github.com/npm/cli/wiki/No-auth-for-URI,-but-auth-present-for-scoped-registry'
const checkResponse =
async ({ method, uri, res, startTime, auth, opts }) => {
opts = { ...defaultOpts, ...opts }
if (res.headers.has('npm-notice') && !res.headers.has('x-local-cache')) {
log.notice('', res.headers.get('npm-notice'))
}
if (res.status >= 400) {
logRequest(method, res, startTime)
if (auth && auth.scopeAuthKey && !auth.token && !auth.auth) {
// we didn't have auth for THIS request, but we do have auth for
// requests to the registry indicated by the spec's scope value.
// Warn the user.
log.warn('registry', `No auth for URI, but auth present for scoped registry.
URI: ${uri}
Scoped Registry Key: ${auth.scopeAuthKey}
More info here: ${moreInfoUrl}`)
}
return checkErrors(method, res, startTime, opts)
} else {
res.body.on('end', () => logRequest(method, res, startTime, opts))
if (opts.ignoreBody) {
res.body.resume()
return new Response(null, res)
}
return res
}
}
module.exports = checkResponse
function logRequest (method, res, startTime) {
const elapsedTime = Date.now() - startTime
const attempt = res.headers.get('x-fetch-attempts')
const attemptStr = attempt && attempt > 1 ? ` attempt #${attempt}` : ''
const cacheStatus = res.headers.get('x-local-cache-status')
const cacheStr = cacheStatus ? ` (cache ${cacheStatus})` : ''
const urlStr = cleanUrl(res.url)
log.http(
'fetch',
`${method.toUpperCase()} ${res.status} ${urlStr} ${elapsedTime}ms${attemptStr}${cacheStr}`
)
}
function checkErrors (method, res, startTime, opts) {
return res.buffer()
.catch(() => null)
.then(body => {
let parsed = body
try {
parsed = JSON.parse(body.toString('utf8'))
} catch {
// ignore errors
}
if (res.status === 401 && res.headers.get('www-authenticate')) {
const auth = res.headers.get('www-authenticate')
.split(/,\s*/)
.map(s => s.toLowerCase())
if (auth.indexOf('ipaddress') !== -1) {
throw new errors.HttpErrorAuthIPAddress(
method, res, parsed, opts.spec
)
} else if (auth.indexOf('otp') !== -1) {
throw new errors.HttpErrorAuthOTP(
method, res, parsed, opts.spec
)
} else {
throw new errors.HttpErrorAuthUnknown(
method, res, parsed, opts.spec
)
}
} else if (
res.status === 401 &&
body != null &&
/one-time pass/.test(body.toString('utf8'))
) {
// Heuristic for malformed OTP responses that don't include the
// www-authenticate header.
throw new errors.HttpErrorAuthOTP(
method, res, parsed, opts.spec
)
} else {
throw new errors.HttpErrorGeneral(
method, res, parsed, opts.spec
)
}
})
}
+26
View File
@@ -0,0 +1,26 @@
const { URL } = require('url')
const replace = '***'
const tokenRegex = /\bnpm_[a-zA-Z0-9]{36}\b/g
const guidRegex = /\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/g
const cleanUrl = (str) => {
if (typeof str !== 'string' || !str) {
return str
}
try {
const url = new URL(str)
if (url.password) {
str = str.replace(url.password, replace)
}
} catch {
// ignore errors
}
return str
.replace(tokenRegex, `npm_${replace}`)
.replace(guidRegex, `npm_${replace}`)
}
module.exports = cleanUrl
+19
View File
@@ -0,0 +1,19 @@
const pkg = require('../package.json')
module.exports = {
maxSockets: 12,
method: 'GET',
registry: 'https://registry.npmjs.org/',
timeout: 5 * 60 * 1000, // 5 minutes
strictSSL: true,
noProxy: process.env.NOPROXY,
userAgent: `${pkg.name
}@${
pkg.version
}/node@${
process.version
}+${
process.arch
} (${
process.platform
})`,
}
+80
View File
@@ -0,0 +1,80 @@
'use strict'
const url = require('url')
function packageName (href) {
try {
let basePath = new url.URL(href).pathname.slice(1)
if (!basePath.match(/^-/)) {
basePath = basePath.split('/')
var index = basePath.indexOf('_rewrite')
if (index === -1) {
index = basePath.length - 1
} else {
index++
}
return decodeURIComponent(basePath[index])
}
} catch (_) {
// this is ok
}
}
class HttpErrorBase extends Error {
constructor (method, res, body, spec) {
super()
this.name = this.constructor.name
this.headers = res.headers.raw()
this.statusCode = res.status
this.code = `E${res.status}`
this.method = method
this.uri = res.url
this.body = body
this.pkgid = spec ? spec.toString() : packageName(res.url)
}
}
module.exports.HttpErrorBase = HttpErrorBase
class HttpErrorGeneral extends HttpErrorBase {
constructor (method, res, body, spec) {
super(method, res, body, spec)
this.message = `${res.status} ${res.statusText} - ${
this.method.toUpperCase()
} ${
this.spec || this.uri
}${
(body && body.error) ? ' - ' + body.error : ''
}`
Error.captureStackTrace(this, HttpErrorGeneral)
}
}
module.exports.HttpErrorGeneral = HttpErrorGeneral
class HttpErrorAuthOTP extends HttpErrorBase {
constructor (method, res, body, spec) {
super(method, res, body, spec)
this.message = 'OTP required for authentication'
this.code = 'EOTP'
Error.captureStackTrace(this, HttpErrorAuthOTP)
}
}
module.exports.HttpErrorAuthOTP = HttpErrorAuthOTP
class HttpErrorAuthIPAddress extends HttpErrorBase {
constructor (method, res, body, spec) {
super(method, res, body, spec)
this.message = 'Login is not allowed from your IP address'
this.code = 'EAUTHIP'
Error.captureStackTrace(this, HttpErrorAuthIPAddress)
}
}
module.exports.HttpErrorAuthIPAddress = HttpErrorAuthIPAddress
class HttpErrorAuthUnknown extends HttpErrorBase {
constructor (method, res, body, spec) {
super(method, res, body, spec)
this.message = 'Unable to authenticate, need: ' + res.headers.get('www-authenticate')
Error.captureStackTrace(this, HttpErrorAuthUnknown)
}
}
module.exports.HttpErrorAuthUnknown = HttpErrorAuthUnknown
+247
View File
@@ -0,0 +1,247 @@
'use strict'
const { HttpErrorAuthOTP } = require('./errors.js')
const checkResponse = require('./check-response.js')
const getAuth = require('./auth.js')
const fetch = require('make-fetch-happen')
const JSONStream = require('minipass-json-stream')
const npa = require('npm-package-arg')
const qs = require('querystring')
const url = require('url')
const zlib = require('minizlib')
const Minipass = require('minipass')
const defaultOpts = require('./default-opts.js')
// WhatWG URL throws if it's not fully resolved
const urlIsValid = u => {
try {
return !!new url.URL(u)
} catch (_) {
return false
}
}
module.exports = regFetch
function regFetch (uri, /* istanbul ignore next */ opts_ = {}) {
const opts = {
...defaultOpts,
...opts_,
}
// if we did not get a fully qualified URI, then we look at the registry
// config or relevant scope to resolve it.
const uriValid = urlIsValid(uri)
let registry = opts.registry || defaultOpts.registry
if (!uriValid) {
registry = opts.registry = (
(opts.spec && pickRegistry(opts.spec, opts)) ||
opts.registry ||
registry
)
uri = `${
registry.trim().replace(/\/?$/g, '')
}/${
uri.trim().replace(/^\//, '')
}`
// asserts that this is now valid
new url.URL(uri)
}
const method = opts.method || 'GET'
// through that takes into account the scope, the prefix of `uri`, etc
const startTime = Date.now()
const auth = getAuth(uri, opts)
const headers = getHeaders(uri, auth, opts)
let body = opts.body
const bodyIsStream = Minipass.isStream(body)
const bodyIsPromise = body &&
typeof body === 'object' &&
typeof body.then === 'function'
if (
body && !bodyIsStream && !bodyIsPromise && typeof body !== 'string' && !Buffer.isBuffer(body)
) {
headers['content-type'] = headers['content-type'] || 'application/json'
body = JSON.stringify(body)
} else if (body && !headers['content-type']) {
headers['content-type'] = 'application/octet-stream'
}
if (opts.gzip) {
headers['content-encoding'] = 'gzip'
if (bodyIsStream) {
const gz = new zlib.Gzip()
body.on('error', /* istanbul ignore next: unlikely and hard to test */
err => gz.emit('error', err))
body = body.pipe(gz)
} else if (!bodyIsPromise) {
body = new zlib.Gzip().end(body).concat()
}
}
const parsed = new url.URL(uri)
if (opts.query) {
const q = typeof opts.query === 'string' ? qs.parse(opts.query)
: opts.query
Object.keys(q).forEach(key => {
if (q[key] !== undefined) {
parsed.searchParams.set(key, q[key])
}
})
uri = url.format(parsed)
}
if (parsed.searchParams.get('write') === 'true' && method === 'GET') {
// do not cache, because this GET is fetching a rev that will be
// used for a subsequent PUT or DELETE, so we need to conditionally
// update cache.
opts.offline = false
opts.preferOffline = false
opts.preferOnline = true
}
const doFetch = async fetchBody => {
const p = fetch(uri, {
agent: opts.agent,
algorithms: opts.algorithms,
body: fetchBody,
cache: getCacheMode(opts),
cachePath: opts.cache,
ca: opts.ca,
cert: auth.cert || opts.cert,
headers,
integrity: opts.integrity,
key: auth.key || opts.key,
localAddress: opts.localAddress,
maxSockets: opts.maxSockets,
memoize: opts.memoize,
method: method,
noProxy: opts.noProxy,
proxy: opts.httpsProxy || opts.proxy,
retry: opts.retry ? opts.retry : {
retries: opts.fetchRetries,
factor: opts.fetchRetryFactor,
minTimeout: opts.fetchRetryMintimeout,
maxTimeout: opts.fetchRetryMaxtimeout,
},
strictSSL: opts.strictSSL,
timeout: opts.timeout || 30 * 1000,
}).then(res => checkResponse({
method,
uri,
res,
registry,
startTime,
auth,
opts,
}))
if (typeof opts.otpPrompt === 'function') {
return p.catch(async er => {
if (er instanceof HttpErrorAuthOTP) {
let otp
// if otp fails to complete, we fail with that failure
try {
otp = await opts.otpPrompt()
} catch (_) {
// ignore this error
}
// if no otp provided, or otpPrompt errored, throw the original HTTP error
if (!otp) {
throw er
}
return regFetch(uri, { ...opts, otp })
}
throw er
})
} else {
return p
}
}
return Promise.resolve(body).then(doFetch)
}
module.exports.json = fetchJSON
function fetchJSON (uri, opts) {
return regFetch(uri, opts).then(res => res.json())
}
module.exports.json.stream = fetchJSONStream
function fetchJSONStream (uri, jsonPath,
/* istanbul ignore next */ opts_ = {}) {
const opts = { ...defaultOpts, ...opts_ }
const parser = JSONStream.parse(jsonPath, opts.mapJSON)
regFetch(uri, opts).then(res =>
res.body.on('error',
/* istanbul ignore next: unlikely and difficult to test */
er => parser.emit('error', er)).pipe(parser)
).catch(er => parser.emit('error', er))
return parser
}
module.exports.pickRegistry = pickRegistry
function pickRegistry (spec, opts = {}) {
spec = npa(spec)
let registry = spec.scope &&
opts[spec.scope.replace(/^@?/, '@') + ':registry']
if (!registry && opts.scope) {
registry = opts[opts.scope.replace(/^@?/, '@') + ':registry']
}
if (!registry) {
registry = opts.registry || defaultOpts.registry
}
return registry
}
function getCacheMode (opts) {
return opts.offline ? 'only-if-cached'
: opts.preferOffline ? 'force-cache'
: opts.preferOnline ? 'no-cache'
: 'default'
}
function getHeaders (uri, auth, opts) {
const headers = Object.assign({
'user-agent': opts.userAgent,
}, opts.headers || {})
if (opts.authType) {
headers['npm-auth-type'] = opts.authType
}
if (opts.scope) {
headers['npm-scope'] = opts.scope
}
if (opts.npmSession) {
headers['npm-session'] = opts.npmSession
}
if (opts.npmCommand) {
headers['npm-command'] = opts.npmCommand
}
// If a tarball is hosted on a different place than the manifest, only send
// credentials on `alwaysAuth`
if (auth.token) {
headers.authorization = `Bearer ${auth.token}`
} else if (auth.auth) {
headers.authorization = `Basic ${auth.auth}`
}
if (opts.otp) {
headers['npm-otp'] = opts.otp
}
return headers
}
module.exports.cleanUrl = require('./clean-url.js')
@@ -0,0 +1,13 @@
Copyright (c) 2015, Rebecca Turner
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
@@ -0,0 +1,133 @@
# hosted-git-info
This will let you identify and transform various git hosts URLs between
protocols. It also can tell you what the URL is for the raw path for
particular file for direct access without git.
## Example
```javascript
const hostedGitInfo = require("hosted-git-info")
const info = hostedGitInfo.fromUrl("git@github.com:npm/hosted-git-info.git", opts)
/* info looks like:
{
type: "github",
domain: "github.com",
user: "npm",
project: "hosted-git-info"
}
*/
```
If the URL can't be matched with a git host, `null` will be returned. We
can match git, ssh and https urls. Additionally, we can match ssh connect
strings (`git@github.com:npm/hosted-git-info`) and shortcuts (eg,
`github:npm/hosted-git-info`). GitHub specifically, is detected in the case
of a third, unprefixed, form: `npm/hosted-git-info`.
If it does match, the returned object has properties of:
* info.type -- The short name of the service
* info.domain -- The domain for git protocol use
* info.user -- The name of the user/org on the git host
* info.project -- The name of the project on the git host
## Version Contract
The major version will be bumped any time…
* The constructor stops accepting URLs that it previously accepted.
* A method is removed.
* A method can no longer accept the number and type of arguments it previously accepted.
* A method can return a different type than it currently returns.
Implications:
* I do not consider the specific format of the urls returned from, say
`.https()` to be a part of the contract. The contract is that it will
return a string that can be used to fetch the repo via HTTPS. But what
that string looks like, specifically, can change.
* Dropping support for a hosted git provider would constitute a breaking
change.
## Usage
### const info = hostedGitInfo.fromUrl(gitSpecifier[, options])
* *gitSpecifer* is a URL of a git repository or a SCP-style specifier of one.
* *options* is an optional object. It can have the following properties:
* *noCommittish* — If true then committishes won't be included in generated URLs.
* *noGitPlus* — If true then `git+` won't be prefixed on URLs.
## Methods
All of the methods take the same options as the `fromUrl` factory. Options
provided to a method override those provided to the constructor.
* info.file(path, opts)
Given the path of a file relative to the repository, returns a URL for
directly fetching it from the githost. If no committish was set then
`HEAD` will be used as the default.
For example `hostedGitInfo.fromUrl("git@github.com:npm/hosted-git-info.git#v1.0.0").file("package.json")`
would return `https://raw.githubusercontent.com/npm/hosted-git-info/v1.0.0/package.json`
* info.shortcut(opts)
eg, `github:npm/hosted-git-info`
* info.browse(path, fragment, opts)
eg, `https://github.com/npm/hosted-git-info/tree/v1.2.0`,
`https://github.com/npm/hosted-git-info/tree/v1.2.0/package.json`,
`https://github.com/npm/hosted-git-info/tree/v1.2.0/REAMDE.md#supported-hosts`
* info.bugs(opts)
eg, `https://github.com/npm/hosted-git-info/issues`
* info.docs(opts)
eg, `https://github.com/npm/hosted-git-info/tree/v1.2.0#readme`
* info.https(opts)
eg, `git+https://github.com/npm/hosted-git-info.git`
* info.sshurl(opts)
eg, `git+ssh://git@github.com/npm/hosted-git-info.git`
* info.ssh(opts)
eg, `git@github.com:npm/hosted-git-info.git`
* info.path(opts)
eg, `npm/hosted-git-info`
* info.tarball(opts)
eg, `https://github.com/npm/hosted-git-info/archive/v1.2.0.tar.gz`
* info.getDefaultRepresentation()
Returns the default output type. The default output type is based on the
string you passed in to be parsed
* info.toString(opts)
Uses the getDefaultRepresentation to call one of the other methods to get a URL for
this resource. As such `hostedGitInfo.fromUrl(url).toString()` will give
you a normalized version of the URL that still uses the same protocol.
Shortcuts will still be returned as shortcuts, but the special case github
form of `org/project` will be normalized to `github:org/project`.
SSH connect strings will be normalized into `git+ssh` URLs.
## Supported hosts
Currently this supports GitHub (including Gists), Bitbucket, GitLab and Sourcehut.
Pull requests for additional hosts welcome.
@@ -0,0 +1,122 @@
'use strict'
const parseUrl = require('./parse-url')
// look for github shorthand inputs, such as npm/cli
const isGitHubShorthand = (arg) => {
// it cannot contain whitespace before the first #
// it cannot start with a / because that's probably an absolute file path
// but it must include a slash since repos are username/repository
// it cannot start with a . because that's probably a relative file path
// it cannot start with an @ because that's a scoped package if it passes the other tests
// it cannot contain a : before a # because that tells us that there's a protocol
// a second / may not exist before a #
const firstHash = arg.indexOf('#')
const firstSlash = arg.indexOf('/')
const secondSlash = arg.indexOf('/', firstSlash + 1)
const firstColon = arg.indexOf(':')
const firstSpace = /\s/.exec(arg)
const firstAt = arg.indexOf('@')
const spaceOnlyAfterHash = !firstSpace || (firstHash > -1 && firstSpace.index > firstHash)
const atOnlyAfterHash = firstAt === -1 || (firstHash > -1 && firstAt > firstHash)
const colonOnlyAfterHash = firstColon === -1 || (firstHash > -1 && firstColon > firstHash)
const secondSlashOnlyAfterHash = secondSlash === -1 || (firstHash > -1 && secondSlash > firstHash)
const hasSlash = firstSlash > 0
// if a # is found, what we really want to know is that the character
// immediately before # is not a /
const doesNotEndWithSlash = firstHash > -1 ? arg[firstHash - 1] !== '/' : !arg.endsWith('/')
const doesNotStartWithDot = !arg.startsWith('.')
return spaceOnlyAfterHash && hasSlash && doesNotEndWithSlash &&
doesNotStartWithDot && atOnlyAfterHash && colonOnlyAfterHash &&
secondSlashOnlyAfterHash
}
module.exports = (giturl, opts, { gitHosts, protocols }) => {
if (!giturl) {
return
}
const correctedUrl = isGitHubShorthand(giturl) ? `github:${giturl}` : giturl
const parsed = parseUrl(correctedUrl, protocols)
if (!parsed) {
return
}
const gitHostShortcut = gitHosts.byShortcut[parsed.protocol]
const gitHostDomain = gitHosts.byDomain[parsed.hostname.startsWith('www.')
? parsed.hostname.slice(4)
: parsed.hostname]
const gitHostName = gitHostShortcut || gitHostDomain
if (!gitHostName) {
return
}
const gitHostInfo = gitHosts[gitHostShortcut || gitHostDomain]
let auth = null
if (protocols[parsed.protocol]?.auth && (parsed.username || parsed.password)) {
auth = `${parsed.username}${parsed.password ? ':' + parsed.password : ''}`
}
let committish = null
let user = null
let project = null
let defaultRepresentation = null
try {
if (gitHostShortcut) {
let pathname = parsed.pathname.startsWith('/') ? parsed.pathname.slice(1) : parsed.pathname
const firstAt = pathname.indexOf('@')
// we ignore auth for shortcuts, so just trim it out
if (firstAt > -1) {
pathname = pathname.slice(firstAt + 1)
}
const lastSlash = pathname.lastIndexOf('/')
if (lastSlash > -1) {
user = decodeURIComponent(pathname.slice(0, lastSlash))
// we want nulls only, never empty strings
if (!user) {
user = null
}
project = decodeURIComponent(pathname.slice(lastSlash + 1))
} else {
project = decodeURIComponent(pathname)
}
if (project.endsWith('.git')) {
project = project.slice(0, -4)
}
if (parsed.hash) {
committish = decodeURIComponent(parsed.hash.slice(1))
}
defaultRepresentation = 'shortcut'
} else {
if (!gitHostInfo.protocols.includes(parsed.protocol)) {
return
}
const segments = gitHostInfo.extract(parsed)
if (!segments) {
return
}
user = segments.user && decodeURIComponent(segments.user)
project = decodeURIComponent(segments.project)
committish = decodeURIComponent(segments.committish)
defaultRepresentation = protocols[parsed.protocol]?.name || parsed.protocol.slice(0, -1)
}
} catch (err) {
/* istanbul ignore else */
if (err instanceof URIError) {
return
} else {
throw err
}
}
return [gitHostName, user, auth, project, committish, defaultRepresentation, opts]
}
@@ -0,0 +1,228 @@
/* eslint-disable max-len */
'use strict'
const maybeJoin = (...args) => args.every(arg => arg) ? args.join('') : ''
const maybeEncode = (arg) => arg ? encodeURIComponent(arg) : ''
const formatHashFragment = (f) => f.toLowerCase().replace(/^\W+|\/|\W+$/g, '').replace(/\W+/g, '-')
const defaults = {
sshtemplate: ({ domain, user, project, committish }) =>
`git@${domain}:${user}/${project}.git${maybeJoin('#', committish)}`,
sshurltemplate: ({ domain, user, project, committish }) =>
`git+ssh://git@${domain}/${user}/${project}.git${maybeJoin('#', committish)}`,
edittemplate: ({ domain, user, project, committish, editpath, path }) =>
`https://${domain}/${user}/${project}${maybeJoin('/', editpath, '/', maybeEncode(committish || 'HEAD'), '/', path)}`,
browsetemplate: ({ domain, user, project, committish, treepath }) =>
`https://${domain}/${user}/${project}${maybeJoin('/', treepath, '/', maybeEncode(committish))}`,
browsetreetemplate: ({ domain, user, project, committish, treepath, path, fragment, hashformat }) =>
`https://${domain}/${user}/${project}/${treepath}/${maybeEncode(committish || 'HEAD')}/${path}${maybeJoin('#', hashformat(fragment || ''))}`,
browseblobtemplate: ({ domain, user, project, committish, blobpath, path, fragment, hashformat }) =>
`https://${domain}/${user}/${project}/${blobpath}/${maybeEncode(committish || 'HEAD')}/${path}${maybeJoin('#', hashformat(fragment || ''))}`,
docstemplate: ({ domain, user, project, treepath, committish }) =>
`https://${domain}/${user}/${project}${maybeJoin('/', treepath, '/', maybeEncode(committish))}#readme`,
httpstemplate: ({ auth, domain, user, project, committish }) =>
`git+https://${maybeJoin(auth, '@')}${domain}/${user}/${project}.git${maybeJoin('#', committish)}`,
filetemplate: ({ domain, user, project, committish, path }) =>
`https://${domain}/${user}/${project}/raw/${maybeEncode(committish || 'HEAD')}/${path}`,
shortcuttemplate: ({ type, user, project, committish }) =>
`${type}:${user}/${project}${maybeJoin('#', committish)}`,
pathtemplate: ({ user, project, committish }) =>
`${user}/${project}${maybeJoin('#', committish)}`,
bugstemplate: ({ domain, user, project }) =>
`https://${domain}/${user}/${project}/issues`,
hashformat: formatHashFragment,
}
const hosts = {}
hosts.github = {
// First two are insecure and generally shouldn't be used any more, but
// they are still supported.
protocols: ['git:', 'http:', 'git+ssh:', 'git+https:', 'ssh:', 'https:'],
domain: 'github.com',
treepath: 'tree',
blobpath: 'blob',
editpath: 'edit',
filetemplate: ({ auth, user, project, committish, path }) =>
`https://${maybeJoin(auth, '@')}raw.githubusercontent.com/${user}/${project}/${maybeEncode(committish || 'HEAD')}/${path}`,
gittemplate: ({ auth, domain, user, project, committish }) =>
`git://${maybeJoin(auth, '@')}${domain}/${user}/${project}.git${maybeJoin('#', committish)}`,
tarballtemplate: ({ domain, user, project, committish }) =>
`https://codeload.${domain}/${user}/${project}/tar.gz/${maybeEncode(committish || 'HEAD')}`,
extract: (url) => {
let [, user, project, type, committish] = url.pathname.split('/', 5)
if (type && type !== 'tree') {
return
}
if (!type) {
committish = url.hash.slice(1)
}
if (project && project.endsWith('.git')) {
project = project.slice(0, -4)
}
if (!user || !project) {
return
}
return { user, project, committish }
},
}
hosts.bitbucket = {
protocols: ['git+ssh:', 'git+https:', 'ssh:', 'https:'],
domain: 'bitbucket.org',
treepath: 'src',
blobpath: 'src',
editpath: '?mode=edit',
edittemplate: ({ domain, user, project, committish, treepath, path, editpath }) =>
`https://${domain}/${user}/${project}${maybeJoin('/', treepath, '/', maybeEncode(committish || 'HEAD'), '/', path, editpath)}`,
tarballtemplate: ({ domain, user, project, committish }) =>
`https://${domain}/${user}/${project}/get/${maybeEncode(committish || 'HEAD')}.tar.gz`,
extract: (url) => {
let [, user, project, aux] = url.pathname.split('/', 4)
if (['get'].includes(aux)) {
return
}
if (project && project.endsWith('.git')) {
project = project.slice(0, -4)
}
if (!user || !project) {
return
}
return { user, project, committish: url.hash.slice(1) }
},
}
hosts.gitlab = {
protocols: ['git+ssh:', 'git+https:', 'ssh:', 'https:'],
domain: 'gitlab.com',
treepath: 'tree',
blobpath: 'tree',
editpath: '-/edit',
httpstemplate: ({ auth, domain, user, project, committish }) =>
`git+https://${maybeJoin(auth, '@')}${domain}/${user}/${project}.git${maybeJoin('#', committish)}`,
tarballtemplate: ({ domain, user, project, committish }) =>
`https://${domain}/${user}/${project}/repository/archive.tar.gz?ref=${maybeEncode(committish || 'HEAD')}`,
extract: (url) => {
const path = url.pathname.slice(1)
if (path.includes('/-/') || path.includes('/archive.tar.gz')) {
return
}
const segments = path.split('/')
let project = segments.pop()
if (project.endsWith('.git')) {
project = project.slice(0, -4)
}
const user = segments.join('/')
if (!user || !project) {
return
}
return { user, project, committish: url.hash.slice(1) }
},
}
hosts.gist = {
protocols: ['git:', 'git+ssh:', 'git+https:', 'ssh:', 'https:'],
domain: 'gist.github.com',
editpath: 'edit',
sshtemplate: ({ domain, project, committish }) =>
`git@${domain}:${project}.git${maybeJoin('#', committish)}`,
sshurltemplate: ({ domain, project, committish }) =>
`git+ssh://git@${domain}/${project}.git${maybeJoin('#', committish)}`,
edittemplate: ({ domain, user, project, committish, editpath }) =>
`https://${domain}/${user}/${project}${maybeJoin('/', maybeEncode(committish))}/${editpath}`,
browsetemplate: ({ domain, project, committish }) =>
`https://${domain}/${project}${maybeJoin('/', maybeEncode(committish))}`,
browsetreetemplate: ({ domain, project, committish, path, hashformat }) =>
`https://${domain}/${project}${maybeJoin('/', maybeEncode(committish))}${maybeJoin('#', hashformat(path))}`,
browseblobtemplate: ({ domain, project, committish, path, hashformat }) =>
`https://${domain}/${project}${maybeJoin('/', maybeEncode(committish))}${maybeJoin('#', hashformat(path))}`,
docstemplate: ({ domain, project, committish }) =>
`https://${domain}/${project}${maybeJoin('/', maybeEncode(committish))}`,
httpstemplate: ({ domain, project, committish }) =>
`git+https://${domain}/${project}.git${maybeJoin('#', committish)}`,
filetemplate: ({ user, project, committish, path }) =>
`https://gist.githubusercontent.com/${user}/${project}/raw${maybeJoin('/', maybeEncode(committish))}/${path}`,
shortcuttemplate: ({ type, project, committish }) =>
`${type}:${project}${maybeJoin('#', committish)}`,
pathtemplate: ({ project, committish }) =>
`${project}${maybeJoin('#', committish)}`,
bugstemplate: ({ domain, project }) =>
`https://${domain}/${project}`,
gittemplate: ({ domain, project, committish }) =>
`git://${domain}/${project}.git${maybeJoin('#', committish)}`,
tarballtemplate: ({ project, committish }) =>
`https://codeload.github.com/gist/${project}/tar.gz/${maybeEncode(committish || 'HEAD')}`,
extract: (url) => {
let [, user, project, aux] = url.pathname.split('/', 4)
if (aux === 'raw') {
return
}
if (!project) {
if (!user) {
return
}
project = user
user = null
}
if (project.endsWith('.git')) {
project = project.slice(0, -4)
}
return { user, project, committish: url.hash.slice(1) }
},
hashformat: function (fragment) {
return fragment && 'file-' + formatHashFragment(fragment)
},
}
hosts.sourcehut = {
protocols: ['git+ssh:', 'https:'],
domain: 'git.sr.ht',
treepath: 'tree',
blobpath: 'tree',
filetemplate: ({ domain, user, project, committish, path }) =>
`https://${domain}/${user}/${project}/blob/${maybeEncode(committish) || 'HEAD'}/${path}`,
httpstemplate: ({ domain, user, project, committish }) =>
`https://${domain}/${user}/${project}.git${maybeJoin('#', committish)}`,
tarballtemplate: ({ domain, user, project, committish }) =>
`https://${domain}/${user}/${project}/archive/${maybeEncode(committish) || 'HEAD'}.tar.gz`,
bugstemplate: ({ user, project }) =>
`https://todo.sr.ht/${user}/${project}`,
extract: (url) => {
let [, user, project, aux] = url.pathname.split('/', 4)
// tarball url
if (['archive'].includes(aux)) {
return
}
if (project && project.endsWith('.git')) {
project = project.slice(0, -4)
}
if (!user || !project) {
return
}
return { user, project, committish: url.hash.slice(1) }
},
}
for (const [name, host] of Object.entries(hosts)) {
hosts[name] = Object.assign({}, defaults, host)
}
module.exports = hosts
@@ -0,0 +1,179 @@
'use strict'
const LRU = require('lru-cache')
const hosts = require('./hosts.js')
const fromUrl = require('./from-url.js')
const parseUrl = require('./parse-url.js')
const cache = new LRU({ max: 1000 })
class GitHost {
constructor (type, user, auth, project, committish, defaultRepresentation, opts = {}) {
Object.assign(this, GitHost.#gitHosts[type], {
type,
user,
auth,
project,
committish,
default: defaultRepresentation,
opts,
})
}
static #gitHosts = { byShortcut: {}, byDomain: {} }
static #protocols = {
'git+ssh:': { name: 'sshurl' },
'ssh:': { name: 'sshurl' },
'git+https:': { name: 'https', auth: true },
'git:': { auth: true },
'http:': { auth: true },
'https:': { auth: true },
'git+http:': { auth: true },
}
static addHost (name, host) {
GitHost.#gitHosts[name] = host
GitHost.#gitHosts.byDomain[host.domain] = name
GitHost.#gitHosts.byShortcut[`${name}:`] = name
GitHost.#protocols[`${name}:`] = { name }
}
static fromUrl (giturl, opts) {
if (typeof giturl !== 'string') {
return
}
const key = giturl + JSON.stringify(opts || {})
if (!cache.has(key)) {
const hostArgs = fromUrl(giturl, opts, {
gitHosts: GitHost.#gitHosts,
protocols: GitHost.#protocols,
})
cache.set(key, hostArgs ? new GitHost(...hostArgs) : undefined)
}
return cache.get(key)
}
static parseUrl (url) {
return parseUrl(url)
}
#fill (template, opts) {
if (typeof template !== 'function') {
return null
}
const options = { ...this, ...this.opts, ...opts }
// the path should always be set so we don't end up with 'undefined' in urls
if (!options.path) {
options.path = ''
}
// template functions will insert the leading slash themselves
if (options.path.startsWith('/')) {
options.path = options.path.slice(1)
}
if (options.noCommittish) {
options.committish = null
}
const result = template(options)
return options.noGitPlus && result.startsWith('git+') ? result.slice(4) : result
}
hash () {
return this.committish ? `#${this.committish}` : ''
}
ssh (opts) {
return this.#fill(this.sshtemplate, opts)
}
sshurl (opts) {
return this.#fill(this.sshurltemplate, opts)
}
browse (path, ...args) {
// not a string, treat path as opts
if (typeof path !== 'string') {
return this.#fill(this.browsetemplate, path)
}
if (typeof args[0] !== 'string') {
return this.#fill(this.browsetreetemplate, { ...args[0], path })
}
return this.#fill(this.browsetreetemplate, { ...args[1], fragment: args[0], path })
}
// If the path is known to be a file, then browseFile should be used. For some hosts
// the url is the same as browse, but for others like GitHub a file can use both `/tree/`
// and `/blob/` in the path. When using a default committish of `HEAD` then the `/tree/`
// path will redirect to a specific commit. Using the `/blob/` path avoids this and
// does not redirect to a different commit.
browseFile (path, ...args) {
if (typeof args[0] !== 'string') {
return this.#fill(this.browseblobtemplate, { ...args[0], path })
}
return this.#fill(this.browseblobtemplate, { ...args[1], fragment: args[0], path })
}
docs (opts) {
return this.#fill(this.docstemplate, opts)
}
bugs (opts) {
return this.#fill(this.bugstemplate, opts)
}
https (opts) {
return this.#fill(this.httpstemplate, opts)
}
git (opts) {
return this.#fill(this.gittemplate, opts)
}
shortcut (opts) {
return this.#fill(this.shortcuttemplate, opts)
}
path (opts) {
return this.#fill(this.pathtemplate, opts)
}
tarball (opts) {
return this.#fill(this.tarballtemplate, { ...opts, noCommittish: false })
}
file (path, opts) {
return this.#fill(this.filetemplate, { ...opts, path })
}
edit (path, opts) {
return this.#fill(this.edittemplate, { ...opts, path })
}
getDefaultRepresentation () {
return this.default
}
toString (opts) {
if (this.default && typeof this[this.default] === 'function') {
return this[this.default](opts)
}
return this.sshurl(opts)
}
}
for (const [name, host] of Object.entries(hosts)) {
GitHost.addHost(name, host)
}
module.exports = GitHost
@@ -0,0 +1,78 @@
const url = require('url')
const lastIndexOfBefore = (str, char, beforeChar) => {
const startPosition = str.indexOf(beforeChar)
return str.lastIndexOf(char, startPosition > -1 ? startPosition : Infinity)
}
const safeUrl = (u) => {
try {
return new url.URL(u)
} catch {
// this fn should never throw
}
}
// accepts input like git:github.com:user/repo and inserts the // after the first :
const correctProtocol = (arg, protocols) => {
const firstColon = arg.indexOf(':')
const proto = arg.slice(0, firstColon + 1)
if (Object.prototype.hasOwnProperty.call(protocols, proto)) {
return arg
}
const firstAt = arg.indexOf('@')
if (firstAt > -1) {
if (firstAt > firstColon) {
return `git+ssh://${arg}`
} else {
return arg
}
}
const doubleSlash = arg.indexOf('//')
if (doubleSlash === firstColon + 1) {
return arg
}
return `${arg.slice(0, firstColon + 1)}//${arg.slice(firstColon + 1)}`
}
// attempt to correct an scp style url so that it will parse with `new URL()`
const correctUrl = (giturl) => {
// ignore @ that come after the first hash since the denotes the start
// of a committish which can contain @ characters
const firstAt = lastIndexOfBefore(giturl, '@', '#')
// ignore colons that come after the hash since that could include colons such as:
// git@github.com:user/package-2#semver:^1.0.0
const lastColonBeforeHash = lastIndexOfBefore(giturl, ':', '#')
if (lastColonBeforeHash > firstAt) {
// the last : comes after the first @ (or there is no @)
// like it would in:
// proto://hostname.com:user/repo
// username@hostname.com:user/repo
// :password@hostname.com:user/repo
// username:password@hostname.com:user/repo
// proto://username@hostname.com:user/repo
// proto://:password@hostname.com:user/repo
// proto://username:password@hostname.com:user/repo
// then we replace the last : with a / to create a valid path
giturl = giturl.slice(0, lastColonBeforeHash) + '/' + giturl.slice(lastColonBeforeHash + 1)
}
if (lastIndexOfBefore(giturl, ':', '#') === -1 && giturl.indexOf('//') === -1) {
// we have no : at all
// as it would be in:
// username@hostname.com/user/repo
// then we prepend a protocol
giturl = `git+ssh://${giturl}`
}
return giturl
}
module.exports = (giturl, protocols) => {
const withProtocol = protocols ? correctProtocol(giturl, protocols) : giturl
return safeUrl(withProtocol) || safeUrl(correctUrl(withProtocol))
}
@@ -0,0 +1,59 @@
{
"name": "hosted-git-info",
"version": "6.1.1",
"description": "Provides metadata and conversions from repository urls for GitHub, Bitbucket and GitLab",
"main": "./lib/index.js",
"repository": {
"type": "git",
"url": "https://github.com/npm/hosted-git-info.git"
},
"keywords": [
"git",
"github",
"bitbucket",
"gitlab"
],
"author": "GitHub Inc.",
"license": "ISC",
"bugs": {
"url": "https://github.com/npm/hosted-git-info/issues"
},
"homepage": "https://github.com/npm/hosted-git-info",
"scripts": {
"posttest": "npm run lint",
"snap": "tap",
"test": "tap",
"test:coverage": "tap --coverage-report=html",
"lint": "eslint \"**/*.js\"",
"postlint": "template-oss-check",
"lintfix": "npm run lint -- --fix",
"template-oss-apply": "template-oss-apply --force"
},
"dependencies": {
"lru-cache": "^7.5.1"
},
"devDependencies": {
"@npmcli/eslint-config": "^4.0.0",
"@npmcli/template-oss": "4.7.1",
"tap": "^16.0.1"
},
"files": [
"bin/",
"lib/"
],
"engines": {
"node": "^14.17.0 || ^16.13.0 || >=18.0.0"
},
"tap": {
"color": 1,
"coverage": true,
"nyc-arg": [
"--exclude",
"tap-snapshots/**"
]
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "4.7.1"
}
}
@@ -0,0 +1,16 @@
ISC License
Copyright 2017-2022 (c) npm, Inc.
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE COPYRIGHT HOLDER DISCLAIMS
ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE
USE OR PERFORMANCE OF THIS SOFTWARE.
@@ -0,0 +1,361 @@
# make-fetch-happen
[![npm version](https://img.shields.io/npm/v/make-fetch-happen.svg)](https://npm.im/make-fetch-happen) [![license](https://img.shields.io/npm/l/make-fetch-happen.svg)](https://npm.im/make-fetch-happen) [![Travis](https://img.shields.io/travis/npm/make-fetch-happen.svg)](https://travis-ci.org/npm/make-fetch-happen) [![Coverage Status](https://coveralls.io/repos/github/npm/make-fetch-happen/badge.svg?branch=latest)](https://coveralls.io/github/npm/make-fetch-happen?branch=latest)
[`make-fetch-happen`](https://github.com/npm/make-fetch-happen) is a Node.js
library that wraps [`minipass-fetch`](https://github.com/npm/minipass-fetch) with additional
features [`minipass-fetch`](https://github.com/npm/minipass-fetch) doesn't intend to include, including HTTP Cache support, request
pooling, proxies, retries, [and more](#features)!
## Install
`$ npm install --save make-fetch-happen`
## Table of Contents
* [Example](#example)
* [Features](#features)
* [Contributing](#contributing)
* [API](#api)
* [`fetch`](#fetch)
* [`fetch.defaults`](#fetch-defaults)
* [`minipass-fetch` options](#minipass-fetch-options)
* [`make-fetch-happen` options](#extra-options)
* [`opts.cachePath`](#opts-cache-path)
* [`opts.cache`](#opts-cache)
* [`opts.proxy`](#opts-proxy)
* [`opts.noProxy`](#opts-no-proxy)
* [`opts.ca, opts.cert, opts.key`](#https-opts)
* [`opts.maxSockets`](#opts-max-sockets)
* [`opts.retry`](#opts-retry)
* [`opts.onRetry`](#opts-onretry)
* [`opts.integrity`](#opts-integrity)
* [`opts.dns`](#opts-dns)
* [Message From Our Sponsors](#wow)
### Example
```javascript
const fetch = require('make-fetch-happen').defaults({
cachePath: './my-cache' // path where cache will be written (and read)
})
fetch('https://registry.npmjs.org/make-fetch-happen').then(res => {
return res.json() // download the body as JSON
}).then(body => {
console.log(`got ${body.name} from web`)
return fetch('https://registry.npmjs.org/make-fetch-happen', {
cache: 'no-cache' // forces a conditional request
})
}).then(res => {
console.log(res.status) // 304! cache validated!
return res.json().then(body => {
console.log(`got ${body.name} from cache`)
})
})
```
### Features
* Builds around [`minipass-fetch`](https://npm.im/minipass-fetch) for the core [`fetch` API](https://fetch.spec.whatwg.org) implementation
* Request pooling out of the box
* Quite fast, really
* Automatic HTTP-semantics-aware request retries
* Cache-fallback automatic "offline mode"
* Proxy support (http, https, socks, socks4, socks5)
* Built-in request caching following full HTTP caching rules (`Cache-Control`, `ETag`, `304`s, cache fallback on error, etc).
* Node.js Stream support
* Transparent gzip and deflate support
* [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) support
* Literally punches nazis
* Built in DNS cache
* (PENDING) Range request caching and resuming
### Contributing
The make-fetch-happen team enthusiastically welcomes contributions and project participation! There's a bunch of things you can do if you want to contribute! The [Contributor Guide](https://github.com/npm/cli/blob/latest/CONTRIBUTING.md) outlines the process for community interaction and contribution. Please don't hesitate to jump in if you'd like to, or even ask us questions if something isn't clear.
All participants and maintainers in this project are expected to follow the [npm Code of Conduct](https://www.npmjs.com/policies/conduct), and just generally be excellent to each other.
Please refer to the [Changelog](CHANGELOG.md) for project history details, too.
Happy hacking!
### API
#### <a name="fetch"></a> `> fetch(uriOrRequest, [opts]) -> Promise<Response>`
This function implements most of the [`fetch` API](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch): given a `uri` string or a `Request` instance, it will fire off an http request and return a Promise containing the relevant response.
If `opts` is provided, the [`minipass-fetch`-specific options](#minipass-fetch-options) will be passed to that library. There are also [additional options](#extra-options) specific to make-fetch-happen that add various features, such as HTTP caching, integrity verification, proxy support, and more.
##### Example
```javascript
fetch('https://google.com').then(res => res.buffer())
```
#### <a name="fetch-defaults"></a> `> fetch.defaults([defaultUrl], [defaultOpts])`
Returns a new `fetch` function that will call `make-fetch-happen` using `defaultUrl` and `defaultOpts` as default values to any calls.
A defaulted `fetch` will also have a `.defaults()` method, so they can be chained.
##### Example
```javascript
const fetch = require('make-fetch-happen').defaults({
cachePath: './my-local-cache'
})
fetch('https://registry.npmjs.org/make-fetch-happen') // will always use the cache
```
#### <a name="minipass-fetch-options"></a> `> minipass-fetch options`
The following options for `minipass-fetch` are used as-is:
* method
* body
* redirect
* follow
* timeout
* compress
* size
These other options are modified or augmented by make-fetch-happen:
* headers - Default `User-Agent` set to make-fetch happen. `Connection` is set to `keep-alive` or `close` automatically depending on `opts.agent`.
* agent
* If agent is null, an http or https Agent will be automatically used. By default, these will be `http.globalAgent` and `https.globalAgent`.
* If [`opts.proxy`](#opts-proxy) is provided and `opts.agent` is null, the agent will be set to an appropriate proxy-handling agent.
* If `opts.agent` is an object, it will be used as the request-pooling agent argument for this request.
* If `opts.agent` is `false`, it will be passed as-is to the underlying request library. This causes a new Agent to be spawned for every request.
For more details, see [the documentation for `minipass-fetch` itself](https://github.com/npm/minipass-fetch#options).
#### <a name="extra-options"></a> `> make-fetch-happen options`
make-fetch-happen augments the `minipass-fetch` API with additional features available through extra options. The following extra options are available:
* [`opts.cachePath`](#opts-cache-path) - Cache target to read/write
* [`opts.cache`](#opts-cache) - `fetch` cache mode. Controls cache *behavior*.
* [`opts.proxy`](#opts-proxy) - Proxy agent
* [`opts.noProxy`](#opts-no-proxy) - Domain segments to disable proxying for.
* [`opts.ca, opts.cert, opts.key, opts.strictSSL`](#https-opts)
* [`opts.localAddress`](#opts-local-address)
* [`opts.maxSockets`](#opts-max-sockets)
* [`opts.retry`](#opts-retry) - Request retry settings
* [`opts.onRetry`](#opts-onretry) - a function called whenever a retry is attempted
* [`opts.integrity`](#opts-integrity) - [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) metadata.
* [`opts.dns`](#opts-dns) - DNS cache options
#### <a name="opts-cache-path"></a> `> opts.cachePath`
A string `Path` to be used as the cache root for [`cacache`](https://npm.im/cacache).
**NOTE**: Requests will not be cached unless their response bodies are consumed. You will need to use one of the `res.json()`, `res.buffer()`, etc methods on the response, or drain the `res.body` stream, in order for it to be written.
The default cache manager also adds the following headers to cached responses:
* `X-Local-Cache`: Path to the cache the content was found in
* `X-Local-Cache-Key`: Unique cache entry key for this response
* `X-Local-Cache-Mode`: Always `stream` to indicate how the response was read from cacache
* `X-Local-Cache-Hash`: Specific integrity hash for the cached entry
* `X-Local-Cache-Status`: One of `miss`, `hit`, `stale`, `revalidated`, `updated`, or `skip` to signal how the response was created
* `X-Local-Cache-Time`: UTCString of the cache insertion time for the entry
Using [`cacache`](https://npm.im/cacache), a call like this may be used to
manually fetch the cached entry:
```javascript
const h = response.headers
cacache.get(h.get('x-local-cache'), h.get('x-local-cache-key'))
// grab content only, directly:
cacache.get.byDigest(h.get('x-local-cache'), h.get('x-local-cache-hash'))
```
##### Example
```javascript
fetch('https://registry.npmjs.org/make-fetch-happen', {
cachePath: './my-local-cache'
}) // -> 200-level response will be written to disk
```
#### <a name="opts-cache"></a> `> opts.cache`
This option follows the standard `fetch` API cache option. This option will do nothing if [`opts.cachePath`](#opts-cache-path) is null. The following values are accepted (as strings):
* `default` - Fetch will inspect the HTTP cache on the way to the network. If there is a fresh response it will be used. If there is a stale response a conditional request will be created, and a normal request otherwise. It then updates the HTTP cache with the response. If the revalidation request fails (for example, on a 500 or if you're offline), the stale response will be returned.
* `no-store` - Fetch behaves as if there is no HTTP cache at all.
* `reload` - Fetch behaves as if there is no HTTP cache on the way to the network. Ergo, it creates a normal request and updates the HTTP cache with the response.
* `no-cache` - Fetch creates a conditional request if there is a response in the HTTP cache and a normal request otherwise. It then updates the HTTP cache with the response.
* `force-cache` - Fetch uses any response in the HTTP cache matching the request, not paying attention to staleness. If there was no response, it creates a normal request and updates the HTTP cache with the response.
* `only-if-cached` - Fetch uses any response in the HTTP cache matching the request, not paying attention to staleness. If there was no response, it returns a network error. (Can only be used when request’s mode is "same-origin". Any cached redirects will be followed assuming request’s redirect mode is "follow" and the redirects do not violate request’s mode.)
(Note: option descriptions are taken from https://fetch.spec.whatwg.org/#http-network-or-cache-fetch)
##### Example
```javascript
const fetch = require('make-fetch-happen').defaults({
cachePath: './my-cache'
})
// Will error with ENOTCACHED if we haven't already cached this url
fetch('https://registry.npmjs.org/make-fetch-happen', {
cache: 'only-if-cached'
})
// Will refresh any local content and cache the new response
fetch('https://registry.npmjs.org/make-fetch-happen', {
cache: 'reload'
})
// Will use any local data, even if stale. Otherwise, will hit network.
fetch('https://registry.npmjs.org/make-fetch-happen', {
cache: 'force-cache'
})
```
#### <a name="opts-proxy"></a> `> opts.proxy`
A string or `new url.URL()`-d URI to proxy through. Different Proxy handlers will be
used depending on the proxy's protocol.
Additionally, `process.env.HTTP_PROXY`, `process.env.HTTPS_PROXY`, and
`process.env.PROXY` are used if present and no `opts.proxy` value is provided.
(Pending) `process.env.NO_PROXY` may also be configured to skip proxying requests for all, or specific domains.
##### Example
```javascript
fetch('https://registry.npmjs.org/make-fetch-happen', {
proxy: 'https://corporate.yourcompany.proxy:4445'
})
fetch('https://registry.npmjs.org/make-fetch-happen', {
proxy: {
protocol: 'https:',
hostname: 'corporate.yourcompany.proxy',
port: 4445
}
})
```
#### <a name="opts-no-proxy"></a> `> opts.noProxy`
If present, should be a comma-separated string or an array of domain extensions
that a proxy should _not_ be used for.
This option may also be provided through `process.env.NO_PROXY`.
#### <a name="https-opts"></a> `> opts.ca, opts.cert, opts.key, opts.strictSSL`
These values are passed in directly to the HTTPS agent and will be used for both
proxied and unproxied outgoing HTTPS requests. They mostly correspond to the
same options the `https` module accepts, which will be themselves passed to
`tls.connect()`. `opts.strictSSL` corresponds to `rejectUnauthorized`.
#### <a name="opts-local-address"></a> `> opts.localAddress`
Passed directly to `http` and `https` request calls. Determines the local
address to bind to.
#### <a name="opts-max-sockets"></a> `> opts.maxSockets`
Default: 15
Maximum number of active concurrent sockets to use for the underlying
Http/Https/Proxy agents. This setting applies once per spawned agent.
15 is probably a _pretty good value_ for most use-cases, and balances speed
with, uh, not knocking out people's routers. 🤓
#### <a name="opts-retry"></a> `> opts.retry`
An object that can be used to tune request retry settings. Retries will only be attempted on the following conditions:
* Request method is NOT `POST` AND
* Request status is one of: `408`, `420`, `429`, or any status in the 500-range. OR
* Request errored with `ECONNRESET`, `ECONNREFUSED`, `EADDRINUSE`, `ETIMEDOUT`, or the `fetch` error `request-timeout`.
The following are worth noting as explicitly not retried:
* `getaddrinfo ENOTFOUND` and will be assumed to be either an unreachable domain or the user will be assumed offline. If a response is cached, it will be returned immediately.
If `opts.retry` is `false`, it is equivalent to `{retries: 0}`
If `opts.retry` is a number, it is equivalent to `{retries: num}`
The following retry options are available if you want more control over it:
* retries
* factor
* minTimeout
* maxTimeout
* randomize
For details on what each of these do, refer to the [`retry`](https://npm.im/retry) documentation.
##### Example
```javascript
fetch('https://flaky.site.com', {
retry: {
retries: 10,
randomize: true
}
})
fetch('http://reliable.site.com', {
retry: false
})
fetch('http://one-more.site.com', {
retry: 3
})
```
#### <a name="opts-onretry"></a> `> opts.onRetry`
A function called with the response or error which caused the retry whenever one is attempted.
##### Example
```javascript
fetch('https://flaky.site.com', {
onRetry(cause) {
console.log('we will retry because of', cause)
}
})
```
#### <a name="opts-integrity"></a> `> opts.integrity`
Matches the response body against the given [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) metadata. If verification fails, the request will fail with an `EINTEGRITY` error.
`integrity` may either be a string or an [`ssri`](https://npm.im/ssri) `Integrity`-like.
##### Example
```javascript
fetch('https://registry.npmjs.org/make-fetch-happen/-/make-fetch-happen-1.0.0.tgz', {
integrity: 'sha1-o47j7zAYnedYFn1dF/fR9OV3z8Q='
}) // -> ok
fetch('https://malicious-registry.org/make-fetch-happen/-/make-fetch-happen-1.0.0.tgz', {
integrity: 'sha1-o47j7zAYnedYFn1dF/fR9OV3z8Q='
}) // Error: EINTEGRITY
```
#### <a name="opts-dns"></a> `> opts.dns`
An object that provides options for the built-in DNS cache. The following options are available:
Note: Due to limitations in the current proxy agent implementation, users of proxies will not benefit from the DNS cache.
* `ttl`: Milliseconds to keep cached DNS responses for. Defaults to `5 * 60 * 1000` (5 minutes)
* `lookup`: A custom lookup function, see [`dns.lookup()`](https://nodejs.org/api/dns.html#dnslookuphostname-options-callback) for implementation details. Defaults to `require('dns').lookup`.
@@ -0,0 +1,214 @@
'use strict'
const LRU = require('lru-cache')
const url = require('url')
const isLambda = require('is-lambda')
const dns = require('./dns.js')
const AGENT_CACHE = new LRU({ max: 50 })
const HttpAgent = require('agentkeepalive')
const HttpsAgent = HttpAgent.HttpsAgent
module.exports = getAgent
const getAgentTimeout = timeout =>
typeof timeout !== 'number' || !timeout ? 0 : timeout + 1
const getMaxSockets = maxSockets => maxSockets || 15
function getAgent (uri, opts) {
const parsedUri = new url.URL(typeof uri === 'string' ? uri : uri.url)
const isHttps = parsedUri.protocol === 'https:'
const pxuri = getProxyUri(parsedUri.href, opts)
// If opts.timeout is zero, set the agentTimeout to zero as well. A timeout
// of zero disables the timeout behavior (OS limits still apply). Else, if
// opts.timeout is a non-zero value, set it to timeout + 1, to ensure that
// the node-fetch-npm timeout will always fire first, giving us more
// consistent errors.
const agentTimeout = getAgentTimeout(opts.timeout)
const agentMaxSockets = getMaxSockets(opts.maxSockets)
const key = [
`https:${isHttps}`,
pxuri
? `proxy:${pxuri.protocol}//${pxuri.host}:${pxuri.port}`
: '>no-proxy<',
`local-address:${opts.localAddress || '>no-local-address<'}`,
`strict-ssl:${isHttps ? opts.rejectUnauthorized : '>no-strict-ssl<'}`,
`ca:${(isHttps && opts.ca) || '>no-ca<'}`,
`cert:${(isHttps && opts.cert) || '>no-cert<'}`,
`key:${(isHttps && opts.key) || '>no-key<'}`,
`timeout:${agentTimeout}`,
`maxSockets:${agentMaxSockets}`,
].join(':')
if (opts.agent != null) { // `agent: false` has special behavior!
return opts.agent
}
// keep alive in AWS lambda makes no sense
const lambdaAgent = !isLambda ? null
: isHttps ? require('https').globalAgent
: require('http').globalAgent
if (isLambda && !pxuri) {
return lambdaAgent
}
if (AGENT_CACHE.peek(key)) {
return AGENT_CACHE.get(key)
}
if (pxuri) {
const pxopts = isLambda ? {
...opts,
agent: lambdaAgent,
} : opts
const proxy = getProxy(pxuri, pxopts, isHttps)
AGENT_CACHE.set(key, proxy)
return proxy
}
const agent = isHttps ? new HttpsAgent({
maxSockets: agentMaxSockets,
ca: opts.ca,
cert: opts.cert,
key: opts.key,
localAddress: opts.localAddress,
rejectUnauthorized: opts.rejectUnauthorized,
timeout: agentTimeout,
freeSocketTimeout: 15000,
lookup: dns.getLookup(opts.dns),
}) : new HttpAgent({
maxSockets: agentMaxSockets,
localAddress: opts.localAddress,
timeout: agentTimeout,
freeSocketTimeout: 15000,
lookup: dns.getLookup(opts.dns),
})
AGENT_CACHE.set(key, agent)
return agent
}
function checkNoProxy (uri, opts) {
const host = new url.URL(uri).hostname.split('.').reverse()
let noproxy = (opts.noProxy || getProcessEnv('no_proxy'))
if (typeof noproxy === 'string') {
noproxy = noproxy.split(',').map(n => n.trim())
}
return noproxy && noproxy.some(no => {
const noParts = no.split('.').filter(x => x).reverse()
if (!noParts.length) {
return false
}
for (let i = 0; i < noParts.length; i++) {
if (host[i] !== noParts[i]) {
return false
}
}
return true
})
}
module.exports.getProcessEnv = getProcessEnv
function getProcessEnv (env) {
if (!env) {
return
}
let value
if (Array.isArray(env)) {
for (const e of env) {
value = process.env[e] ||
process.env[e.toUpperCase()] ||
process.env[e.toLowerCase()]
if (typeof value !== 'undefined') {
break
}
}
}
if (typeof env === 'string') {
value = process.env[env] ||
process.env[env.toUpperCase()] ||
process.env[env.toLowerCase()]
}
return value
}
module.exports.getProxyUri = getProxyUri
function getProxyUri (uri, opts) {
const protocol = new url.URL(uri).protocol
const proxy = opts.proxy ||
(
protocol === 'https:' &&
getProcessEnv('https_proxy')
) ||
(
protocol === 'http:' &&
getProcessEnv(['https_proxy', 'http_proxy', 'proxy'])
)
if (!proxy) {
return null
}
const parsedProxy = (typeof proxy === 'string') ? new url.URL(proxy) : proxy
return !checkNoProxy(uri, opts) && parsedProxy
}
const getAuth = u =>
u.username && u.password ? decodeURIComponent(`${u.username}:${u.password}`)
: u.username ? decodeURIComponent(u.username)
: null
const getPath = u => u.pathname + u.search + u.hash
const HttpProxyAgent = require('http-proxy-agent')
const HttpsProxyAgent = require('https-proxy-agent')
const { SocksProxyAgent } = require('socks-proxy-agent')
module.exports.getProxy = getProxy
function getProxy (proxyUrl, opts, isHttps) {
// our current proxy agents do not support an overridden dns lookup method, so will not
// benefit from the dns cache
const popts = {
host: proxyUrl.hostname,
port: proxyUrl.port,
protocol: proxyUrl.protocol,
path: getPath(proxyUrl),
auth: getAuth(proxyUrl),
ca: opts.ca,
cert: opts.cert,
key: opts.key,
timeout: getAgentTimeout(opts.timeout),
localAddress: opts.localAddress,
maxSockets: getMaxSockets(opts.maxSockets),
rejectUnauthorized: opts.rejectUnauthorized,
}
if (proxyUrl.protocol === 'http:' || proxyUrl.protocol === 'https:') {
if (!isHttps) {
return new HttpProxyAgent(popts)
} else {
return new HttpsProxyAgent(popts)
}
} else if (proxyUrl.protocol.startsWith('socks')) {
// socks-proxy-agent uses hostname not host
popts.hostname = popts.host
delete popts.host
return new SocksProxyAgent(popts)
} else {
throw Object.assign(
new Error(`unsupported proxy protocol: '${proxyUrl.protocol}'`),
{
code: 'EUNSUPPORTEDPROXY',
url: proxyUrl.href,
}
)
}
}
@@ -0,0 +1,444 @@
const { Request, Response } = require('minipass-fetch')
const Minipass = require('minipass')
const MinipassFlush = require('minipass-flush')
const cacache = require('cacache')
const url = require('url')
const CachingMinipassPipeline = require('../pipeline.js')
const CachePolicy = require('./policy.js')
const cacheKey = require('./key.js')
const remote = require('../remote.js')
const hasOwnProperty = (obj, prop) => Object.prototype.hasOwnProperty.call(obj, prop)
// allow list for request headers that will be written to the cache index
// note: we will also store any request headers
// that are named in a response's vary header
const KEEP_REQUEST_HEADERS = [
'accept-charset',
'accept-encoding',
'accept-language',
'accept',
'cache-control',
]
// allow list for response headers that will be written to the cache index
// note: we must not store the real response's age header, or when we load
// a cache policy based on the metadata it will think the cached response
// is always stale
const KEEP_RESPONSE_HEADERS = [
'cache-control',
'content-encoding',
'content-language',
'content-type',
'date',
'etag',
'expires',
'last-modified',
'link',
'location',
'pragma',
'vary',
]
// return an object containing all metadata to be written to the index
const getMetadata = (request, response, options) => {
const metadata = {
time: Date.now(),
url: request.url,
reqHeaders: {},
resHeaders: {},
// options on which we must match the request and vary the response
options: {
compress: options.compress != null ? options.compress : request.compress,
},
}
// only save the status if it's not a 200 or 304
if (response.status !== 200 && response.status !== 304) {
metadata.status = response.status
}
for (const name of KEEP_REQUEST_HEADERS) {
if (request.headers.has(name)) {
metadata.reqHeaders[name] = request.headers.get(name)
}
}
// if the request's host header differs from the host in the url
// we need to keep it, otherwise it's just noise and we ignore it
const host = request.headers.get('host')
const parsedUrl = new url.URL(request.url)
if (host && parsedUrl.host !== host) {
metadata.reqHeaders.host = host
}
// if the response has a vary header, make sure
// we store the relevant request headers too
if (response.headers.has('vary')) {
const vary = response.headers.get('vary')
// a vary of "*" means every header causes a different response.
// in that scenario, we do not include any additional headers
// as the freshness check will always fail anyway and we don't
// want to bloat the cache indexes
if (vary !== '*') {
// copy any other request headers that will vary the response
const varyHeaders = vary.trim().toLowerCase().split(/\s*,\s*/)
for (const name of varyHeaders) {
if (request.headers.has(name)) {
metadata.reqHeaders[name] = request.headers.get(name)
}
}
}
}
for (const name of KEEP_RESPONSE_HEADERS) {
if (response.headers.has(name)) {
metadata.resHeaders[name] = response.headers.get(name)
}
}
return metadata
}
// symbols used to hide objects that may be lazily evaluated in a getter
const _request = Symbol('request')
const _response = Symbol('response')
const _policy = Symbol('policy')
class CacheEntry {
constructor ({ entry, request, response, options }) {
if (entry) {
this.key = entry.key
this.entry = entry
// previous versions of this module didn't write an explicit timestamp in
// the metadata, so fall back to the entry's timestamp. we can't use the
// entry timestamp to determine staleness because cacache will update it
// when it verifies its data
this.entry.metadata.time = this.entry.metadata.time || this.entry.time
} else {
this.key = cacheKey(request)
}
this.options = options
// these properties are behind getters that lazily evaluate
this[_request] = request
this[_response] = response
this[_policy] = null
}
// returns a CacheEntry instance that satisfies the given request
// or undefined if no existing entry satisfies
static async find (request, options) {
try {
// compacts the index and returns an array of unique entries
var matches = await cacache.index.compact(options.cachePath, cacheKey(request), (A, B) => {
const entryA = new CacheEntry({ entry: A, options })
const entryB = new CacheEntry({ entry: B, options })
return entryA.policy.satisfies(entryB.request)
}, {
validateEntry: (entry) => {
// clean out entries with a buggy content-encoding value
if (entry.metadata &&
entry.metadata.resHeaders &&
entry.metadata.resHeaders['content-encoding'] === null) {
return false
}
// if an integrity is null, it needs to have a status specified
if (entry.integrity === null) {
return !!(entry.metadata && entry.metadata.status)
}
return true
},
})
} catch (err) {
// if the compact request fails, ignore the error and return
return
}
// a cache mode of 'reload' means to behave as though we have no cache
// on the way to the network. return undefined to allow cacheFetch to
// create a brand new request no matter what.
if (options.cache === 'reload') {
return
}
// find the specific entry that satisfies the request
let match
for (const entry of matches) {
const _entry = new CacheEntry({
entry,
options,
})
if (_entry.policy.satisfies(request)) {
match = _entry
break
}
}
return match
}
// if the user made a PUT/POST/PATCH then we invalidate our
// cache for the same url by deleting the index entirely
static async invalidate (request, options) {
const key = cacheKey(request)
try {
await cacache.rm.entry(options.cachePath, key, { removeFully: true })
} catch (err) {
// ignore errors
}
}
get request () {
if (!this[_request]) {
this[_request] = new Request(this.entry.metadata.url, {
method: 'GET',
headers: this.entry.metadata.reqHeaders,
...this.entry.metadata.options,
})
}
return this[_request]
}
get response () {
if (!this[_response]) {
this[_response] = new Response(null, {
url: this.entry.metadata.url,
counter: this.options.counter,
status: this.entry.metadata.status || 200,
headers: {
...this.entry.metadata.resHeaders,
'content-length': this.entry.size,
},
})
}
return this[_response]
}
get policy () {
if (!this[_policy]) {
this[_policy] = new CachePolicy({
entry: this.entry,
request: this.request,
response: this.response,
options: this.options,
})
}
return this[_policy]
}
// wraps the response in a pipeline that stores the data
// in the cache while the user consumes it
async store (status) {
// if we got a status other than 200, 301, or 308,
// or the CachePolicy forbid storage, append the
// cache status header and return it untouched
if (
this.request.method !== 'GET' ||
![200, 301, 308].includes(this.response.status) ||
!this.policy.storable()
) {
this.response.headers.set('x-local-cache-status', 'skip')
return this.response
}
const size = this.response.headers.get('content-length')
const cacheOpts = {
algorithms: this.options.algorithms,
metadata: getMetadata(this.request, this.response, this.options),
size,
integrity: this.options.integrity,
integrityEmitter: this.response.body.hasIntegrityEmitter && this.response.body,
}
let body = null
// we only set a body if the status is a 200, redirects are
// stored as metadata only
if (this.response.status === 200) {
let cacheWriteResolve, cacheWriteReject
const cacheWritePromise = new Promise((resolve, reject) => {
cacheWriteResolve = resolve
cacheWriteReject = reject
})
body = new CachingMinipassPipeline({ events: ['integrity', 'size'] }, new MinipassFlush({
flush () {
return cacheWritePromise
},
}))
// this is always true since if we aren't reusing the one from the remote fetch, we
// are using the one from cacache
body.hasIntegrityEmitter = true
const onResume = () => {
const tee = new Minipass()
const cacheStream = cacache.put.stream(this.options.cachePath, this.key, cacheOpts)
// re-emit the integrity and size events on our new response body so they can be reused
cacheStream.on('integrity', i => body.emit('integrity', i))
cacheStream.on('size', s => body.emit('size', s))
// stick a flag on here so downstream users will know if they can expect integrity events
tee.pipe(cacheStream)
// TODO if the cache write fails, log a warning but return the response anyway
// eslint-disable-next-line promise/catch-or-return
cacheStream.promise().then(cacheWriteResolve, cacheWriteReject)
body.unshift(tee)
body.unshift(this.response.body)
}
body.once('resume', onResume)
body.once('end', () => body.removeListener('resume', onResume))
} else {
await cacache.index.insert(this.options.cachePath, this.key, null, cacheOpts)
}
// note: we do not set the x-local-cache-hash header because we do not know
// the hash value until after the write to the cache completes, which doesn't
// happen until after the response has been sent and it's too late to write
// the header anyway
this.response.headers.set('x-local-cache', encodeURIComponent(this.options.cachePath))
this.response.headers.set('x-local-cache-key', encodeURIComponent(this.key))
this.response.headers.set('x-local-cache-mode', 'stream')
this.response.headers.set('x-local-cache-status', status)
this.response.headers.set('x-local-cache-time', new Date().toISOString())
const newResponse = new Response(body, {
url: this.response.url,
status: this.response.status,
headers: this.response.headers,
counter: this.options.counter,
})
return newResponse
}
// use the cached data to create a response and return it
async respond (method, options, status) {
let response
if (method === 'HEAD' || [301, 308].includes(this.response.status)) {
// if the request is a HEAD, or the response is a redirect,
// then the metadata in the entry already includes everything
// we need to build a response
response = this.response
} else {
// we're responding with a full cached response, so create a body
// that reads from cacache and attach it to a new Response
const body = new Minipass()
const headers = { ...this.policy.responseHeaders() }
const onResume = () => {
const cacheStream = cacache.get.stream.byDigest(
this.options.cachePath, this.entry.integrity, { memoize: this.options.memoize }
)
cacheStream.on('error', async (err) => {
cacheStream.pause()
if (err.code === 'EINTEGRITY') {
await cacache.rm.content(
this.options.cachePath, this.entry.integrity, { memoize: this.options.memoize }
)
}
if (err.code === 'ENOENT' || err.code === 'EINTEGRITY') {
await CacheEntry.invalidate(this.request, this.options)
}
body.emit('error', err)
cacheStream.resume()
})
// emit the integrity and size events based on our metadata so we're consistent
body.emit('integrity', this.entry.integrity)
body.emit('size', Number(headers['content-length']))
cacheStream.pipe(body)
}
body.once('resume', onResume)
body.once('end', () => body.removeListener('resume', onResume))
response = new Response(body, {
url: this.entry.metadata.url,
counter: options.counter,
status: 200,
headers,
})
}
response.headers.set('x-local-cache', encodeURIComponent(this.options.cachePath))
response.headers.set('x-local-cache-hash', encodeURIComponent(this.entry.integrity))
response.headers.set('x-local-cache-key', encodeURIComponent(this.key))
response.headers.set('x-local-cache-mode', 'stream')
response.headers.set('x-local-cache-status', status)
response.headers.set('x-local-cache-time', new Date(this.entry.metadata.time).toUTCString())
return response
}
// use the provided request along with this cache entry to
// revalidate the stored response. returns a response, either
// from the cache or from the update
async revalidate (request, options) {
const revalidateRequest = new Request(request, {
headers: this.policy.revalidationHeaders(request),
})
try {
// NOTE: be sure to remove the headers property from the
// user supplied options, since we have already defined
// them on the new request object. if they're still in the
// options then those will overwrite the ones from the policy
var response = await remote(revalidateRequest, {
...options,
headers: undefined,
})
} catch (err) {
// if the network fetch fails, return the stale
// cached response unless it has a cache-control
// of 'must-revalidate'
if (!this.policy.mustRevalidate) {
return this.respond(request.method, options, 'stale')
}
throw err
}
if (this.policy.revalidated(revalidateRequest, response)) {
// we got a 304, write a new index to the cache and respond from cache
const metadata = getMetadata(request, response, options)
// 304 responses do not include headers that are specific to the response data
// since they do not include a body, so we copy values for headers that were
// in the old cache entry to the new one, if the new metadata does not already
// include that header
for (const name of KEEP_RESPONSE_HEADERS) {
if (
!hasOwnProperty(metadata.resHeaders, name) &&
hasOwnProperty(this.entry.metadata.resHeaders, name)
) {
metadata.resHeaders[name] = this.entry.metadata.resHeaders[name]
}
}
try {
await cacache.index.insert(options.cachePath, this.key, this.entry.integrity, {
size: this.entry.size,
metadata,
})
} catch (err) {
// if updating the cache index fails, we ignore it and
// respond anyway
}
return this.respond(request.method, options, 'revalidated')
}
// if we got a modified response, create a new entry based on it
const newEntry = new CacheEntry({
request,
response,
options,
})
// respond with the new entry while writing it to the cache
return newEntry.store('updated')
}
}
module.exports = CacheEntry
@@ -0,0 +1,11 @@
class NotCachedError extends Error {
constructor (url) {
/* eslint-disable-next-line max-len */
super(`request to ${url} failed: cache mode is 'only-if-cached' but no cached response is available.`)
this.code = 'ENOTCACHED'
}
}
module.exports = {
NotCachedError,
}
@@ -0,0 +1,49 @@
const { NotCachedError } = require('./errors.js')
const CacheEntry = require('./entry.js')
const remote = require('../remote.js')
// do whatever is necessary to get a Response and return it
const cacheFetch = async (request, options) => {
// try to find a cached entry that satisfies this request
const entry = await CacheEntry.find(request, options)
if (!entry) {
// no cached result, if the cache mode is 'only-if-cached' that's a failure
if (options.cache === 'only-if-cached') {
throw new NotCachedError(request.url)
}
// otherwise, we make a request, store it and return it
const response = await remote(request, options)
const newEntry = new CacheEntry({ request, response, options })
return newEntry.store('miss')
}
// we have a cached response that satisfies this request, however if the cache
// mode is 'no-cache' then we send the revalidation request no matter what
if (options.cache === 'no-cache') {
return entry.revalidate(request, options)
}
// if the cached entry is not stale, or if the cache mode is 'force-cache' or
// 'only-if-cached' we can respond with the cached entry. set the status
// based on the result of needsRevalidation and respond
const _needsRevalidation = entry.policy.needsRevalidation(request)
if (options.cache === 'force-cache' ||
options.cache === 'only-if-cached' ||
!_needsRevalidation) {
return entry.respond(request.method, options, _needsRevalidation ? 'stale' : 'hit')
}
// if we got here, the cache entry is stale so revalidate it
return entry.revalidate(request, options)
}
cacheFetch.invalidate = async (request, options) => {
if (!options.cachePath) {
return
}
return CacheEntry.invalidate(request, options)
}
module.exports = cacheFetch
@@ -0,0 +1,17 @@
const { URL, format } = require('url')
// options passed to url.format() when generating a key
const formatOptions = {
auth: false,
fragment: false,
search: true,
unicode: false,
}
// returns a string to be used as the cache key for the Request
const cacheKey = (request) => {
const parsed = new URL(request.url)
return `make-fetch-happen:request-cache:${format(parsed, formatOptions)}`
}
module.exports = cacheKey
@@ -0,0 +1,161 @@
const CacheSemantics = require('http-cache-semantics')
const Negotiator = require('negotiator')
const ssri = require('ssri')
// options passed to http-cache-semantics constructor
const policyOptions = {
shared: false,
ignoreCargoCult: true,
}
// a fake empty response, used when only testing the
// request for storability
const emptyResponse = { status: 200, headers: {} }
// returns a plain object representation of the Request
const requestObject = (request) => {
const _obj = {
method: request.method,
url: request.url,
headers: {},
compress: request.compress,
}
request.headers.forEach((value, key) => {
_obj.headers[key] = value
})
return _obj
}
// returns a plain object representation of the Response
const responseObject = (response) => {
const _obj = {
status: response.status,
headers: {},
}
response.headers.forEach((value, key) => {
_obj.headers[key] = value
})
return _obj
}
class CachePolicy {
constructor ({ entry, request, response, options }) {
this.entry = entry
this.request = requestObject(request)
this.response = responseObject(response)
this.options = options
this.policy = new CacheSemantics(this.request, this.response, policyOptions)
if (this.entry) {
// if we have an entry, copy the timestamp to the _responseTime
// this is necessary because the CacheSemantics constructor forces
// the value to Date.now() which means a policy created from a
// cache entry is likely to always identify itself as stale
this.policy._responseTime = this.entry.metadata.time
}
}
// static method to quickly determine if a request alone is storable
static storable (request, options) {
// no cachePath means no caching
if (!options.cachePath) {
return false
}
// user explicitly asked not to cache
if (options.cache === 'no-store') {
return false
}
// we only cache GET and HEAD requests
if (!['GET', 'HEAD'].includes(request.method)) {
return false
}
// otherwise, let http-cache-semantics make the decision
// based on the request's headers
const policy = new CacheSemantics(requestObject(request), emptyResponse, policyOptions)
return policy.storable()
}
// returns true if the policy satisfies the request
satisfies (request) {
const _req = requestObject(request)
if (this.request.headers.host !== _req.headers.host) {
return false
}
if (this.request.compress !== _req.compress) {
return false
}
const negotiatorA = new Negotiator(this.request)
const negotiatorB = new Negotiator(_req)
if (JSON.stringify(negotiatorA.mediaTypes()) !== JSON.stringify(negotiatorB.mediaTypes())) {
return false
}
if (JSON.stringify(negotiatorA.languages()) !== JSON.stringify(negotiatorB.languages())) {
return false
}
if (JSON.stringify(negotiatorA.encodings()) !== JSON.stringify(negotiatorB.encodings())) {
return false
}
if (this.options.integrity) {
return ssri.parse(this.options.integrity).match(this.entry.integrity)
}
return true
}
// returns true if the request and response allow caching
storable () {
return this.policy.storable()
}
// NOTE: this is a hack to avoid parsing the cache-control
// header ourselves, it returns true if the response's
// cache-control contains must-revalidate
get mustRevalidate () {
return !!this.policy._rescc['must-revalidate']
}
// returns true if the cached response requires revalidation
// for the given request
needsRevalidation (request) {
const _req = requestObject(request)
// force method to GET because we only cache GETs
// but can serve a HEAD from a cached GET
_req.method = 'GET'
return !this.policy.satisfiesWithoutRevalidation(_req)
}
responseHeaders () {
return this.policy.responseHeaders()
}
// returns a new object containing the appropriate headers
// to send a revalidation request
revalidationHeaders (request) {
const _req = requestObject(request)
return this.policy.revalidationHeaders(_req)
}
// returns true if the request/response was revalidated
// successfully. returns false if a new response was received
revalidated (request, response) {
const _req = requestObject(request)
const _res = responseObject(response)
const policy = this.policy.revalidatedPolicy(_req, _res)
return !policy.modified
}
}
module.exports = CachePolicy
@@ -0,0 +1,49 @@
const LRUCache = require('lru-cache')
const dns = require('dns')
const defaultOptions = exports.defaultOptions = {
family: undefined,
hints: dns.ADDRCONFIG,
all: false,
verbatim: undefined,
}
const lookupCache = exports.lookupCache = new LRUCache({ max: 50 })
// this is a factory so that each request can have its own opts (i.e. ttl)
// while still sharing the cache across all requests
exports.getLookup = (dnsOptions) => {
return (hostname, options, callback) => {
if (typeof options === 'function') {
callback = options
options = null
} else if (typeof options === 'number') {
options = { family: options }
}
options = { ...defaultOptions, ...options }
const key = JSON.stringify({
hostname,
family: options.family,
hints: options.hints,
all: options.all,
verbatim: options.verbatim,
})
if (lookupCache.has(key)) {
const [address, family] = lookupCache.get(key)
process.nextTick(callback, null, address, family)
return
}
dnsOptions.lookup(hostname, options, (err, address, family) => {
if (err) {
return callback(err)
}
lookupCache.set(key, [address, family], { ttl: dnsOptions.ttl })
return callback(null, address, family)
})
}
}
@@ -0,0 +1,118 @@
'use strict'
const { FetchError, Request, isRedirect } = require('minipass-fetch')
const url = require('url')
const CachePolicy = require('./cache/policy.js')
const cache = require('./cache/index.js')
const remote = require('./remote.js')
// given a Request, a Response and user options
// return true if the response is a redirect that
// can be followed. we throw errors that will result
// in the fetch being rejected if the redirect is
// possible but invalid for some reason
const canFollowRedirect = (request, response, options) => {
if (!isRedirect(response.status)) {
return false
}
if (options.redirect === 'manual') {
return false
}
if (options.redirect === 'error') {
throw new FetchError(`redirect mode is set to error: ${request.url}`,
'no-redirect', { code: 'ENOREDIRECT' })
}
if (!response.headers.has('location')) {
throw new FetchError(`redirect location header missing for: ${request.url}`,
'no-location', { code: 'EINVALIDREDIRECT' })
}
if (request.counter >= request.follow) {
throw new FetchError(`maximum redirect reached at: ${request.url}`,
'max-redirect', { code: 'EMAXREDIRECT' })
}
return true
}
// given a Request, a Response, and the user's options return an object
// with a new Request and a new options object that will be used for
// following the redirect
const getRedirect = (request, response, options) => {
const _opts = { ...options }
const location = response.headers.get('location')
const redirectUrl = new url.URL(location, /^https?:/.test(location) ? undefined : request.url)
// Comment below is used under the following license:
/**
* @license
* Copyright (c) 2010-2012 Mikeal Rogers
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an "AS
* IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language
* governing permissions and limitations under the License.
*/
// Remove authorization if changing hostnames (but not if just
// changing ports or protocols). This matches the behavior of request:
// https://github.com/request/request/blob/b12a6245/lib/redirect.js#L134-L138
if (new url.URL(request.url).hostname !== redirectUrl.hostname) {
request.headers.delete('authorization')
request.headers.delete('cookie')
}
// for POST request with 301/302 response, or any request with 303 response,
// use GET when following redirect
if (
response.status === 303 ||
(request.method === 'POST' && [301, 302].includes(response.status))
) {
_opts.method = 'GET'
_opts.body = null
request.headers.delete('content-length')
}
_opts.headers = {}
request.headers.forEach((value, key) => {
_opts.headers[key] = value
})
_opts.counter = ++request.counter
const redirectReq = new Request(url.format(redirectUrl), _opts)
return {
request: redirectReq,
options: _opts,
}
}
const fetch = async (request, options) => {
const response = CachePolicy.storable(request, options)
? await cache(request, options)
: await remote(request, options)
// if the request wasn't a GET or HEAD, and the response
// status is between 200 and 399 inclusive, invalidate the
// request url
if (!['GET', 'HEAD'].includes(request.method) &&
response.status >= 200 &&
response.status <= 399) {
await cache.invalidate(request, options)
}
if (!canFollowRedirect(request, response, options)) {
return response
}
const redirect = getRedirect(request, response, options)
return fetch(redirect.request, redirect.options)
}
module.exports = fetch
@@ -0,0 +1,41 @@
const { FetchError, Headers, Request, Response } = require('minipass-fetch')
const configureOptions = require('./options.js')
const fetch = require('./fetch.js')
const makeFetchHappen = (url, opts) => {
const options = configureOptions(opts)
const request = new Request(url, options)
return fetch(request, options)
}
makeFetchHappen.defaults = (defaultUrl, defaultOptions = {}, wrappedFetch = makeFetchHappen) => {
if (typeof defaultUrl === 'object') {
defaultOptions = defaultUrl
defaultUrl = null
}
const defaultedFetch = (url, options = {}) => {
const finalUrl = url || defaultUrl
const finalOptions = {
...defaultOptions,
...options,
headers: {
...defaultOptions.headers,
...options.headers,
},
}
return wrappedFetch(finalUrl, finalOptions)
}
defaultedFetch.defaults = (defaultUrl1, defaultOptions1 = {}) =>
makeFetchHappen.defaults(defaultUrl1, defaultOptions1, defaultedFetch)
return defaultedFetch
}
module.exports = makeFetchHappen
module.exports.FetchError = FetchError
module.exports.Headers = Headers
module.exports.Request = Request
module.exports.Response = Response
@@ -0,0 +1,52 @@
const dns = require('dns')
const conditionalHeaders = [
'if-modified-since',
'if-none-match',
'if-unmodified-since',
'if-match',
'if-range',
]
const configureOptions = (opts) => {
const { strictSSL, ...options } = { ...opts }
options.method = options.method ? options.method.toUpperCase() : 'GET'
options.rejectUnauthorized = strictSSL !== false
if (!options.retry) {
options.retry = { retries: 0 }
} else if (typeof options.retry === 'string') {
const retries = parseInt(options.retry, 10)
if (isFinite(retries)) {
options.retry = { retries }
} else {
options.retry = { retries: 0 }
}
} else if (typeof options.retry === 'number') {
options.retry = { retries: options.retry }
} else {
options.retry = { retries: 0, ...options.retry }
}
options.dns = { ttl: 5 * 60 * 1000, lookup: dns.lookup, ...options.dns }
options.cache = options.cache || 'default'
if (options.cache === 'default') {
const hasConditionalHeader = Object.keys(options.headers || {}).some((name) => {
return conditionalHeaders.includes(name.toLowerCase())
})
if (hasConditionalHeader) {
options.cache = 'no-store'
}
}
// cacheManager is deprecated, but if it's set and
// cachePath is not we should copy it to the new field
if (options.cacheManager && !options.cachePath) {
options.cachePath = options.cacheManager
}
return options
}
module.exports = configureOptions
@@ -0,0 +1,41 @@
'use strict'
const MinipassPipeline = require('minipass-pipeline')
class CachingMinipassPipeline extends MinipassPipeline {
#events = []
#data = new Map()
constructor (opts, ...streams) {
// CRITICAL: do NOT pass the streams to the call to super(), this will start
// the flow of data and potentially cause the events we need to catch to emit
// before we've finished our own setup. instead we call super() with no args,
// finish our setup, and then push the streams into ourselves to start the
// data flow
super()
this.#events = opts.events
/* istanbul ignore next - coverage disabled because this is pointless to test here */
if (streams.length) {
this.push(...streams)
}
}
on (event, handler) {
if (this.#events.includes(event) && this.#data.has(event)) {
return handler(...this.#data.get(event))
}
return super.on(event, handler)
}
emit (event, ...data) {
if (this.#events.includes(event)) {
this.#data.set(event, data)
}
return super.emit(event, ...data)
}
}
module.exports = CachingMinipassPipeline
@@ -0,0 +1,121 @@
const Minipass = require('minipass')
const fetch = require('minipass-fetch')
const promiseRetry = require('promise-retry')
const ssri = require('ssri')
const CachingMinipassPipeline = require('./pipeline.js')
const getAgent = require('./agent.js')
const pkg = require('../package.json')
const USER_AGENT = `${pkg.name}/${pkg.version} (+https://npm.im/${pkg.name})`
const RETRY_ERRORS = [
'ECONNRESET', // remote socket closed on us
'ECONNREFUSED', // remote host refused to open connection
'EADDRINUSE', // failed to bind to a local port (proxy?)
'ETIMEDOUT', // someone in the transaction is WAY TOO SLOW
'ERR_SOCKET_TIMEOUT', // same as above, but this one comes from agentkeepalive
// Known codes we do NOT retry on:
// ENOTFOUND (getaddrinfo failure. Either bad hostname, or offline)
]
const RETRY_TYPES = [
'request-timeout',
]
// make a request directly to the remote source,
// retrying certain classes of errors as well as
// following redirects (through the cache if necessary)
// and verifying response integrity
const remoteFetch = (request, options) => {
const agent = getAgent(request.url, options)
if (!request.headers.has('connection')) {
request.headers.set('connection', agent ? 'keep-alive' : 'close')
}
if (!request.headers.has('user-agent')) {
request.headers.set('user-agent', USER_AGENT)
}
// keep our own options since we're overriding the agent
// and the redirect mode
const _opts = {
...options,
agent,
redirect: 'manual',
}
return promiseRetry(async (retryHandler, attemptNum) => {
const req = new fetch.Request(request, _opts)
try {
let res = await fetch(req, _opts)
if (_opts.integrity && res.status === 200) {
// we got a 200 response and the user has specified an expected
// integrity value, so wrap the response in an ssri stream to verify it
const integrityStream = ssri.integrityStream({
algorithms: _opts.algorithms,
integrity: _opts.integrity,
size: _opts.size,
})
const pipeline = new CachingMinipassPipeline({
events: ['integrity', 'size'],
}, res.body, integrityStream)
// we also propagate the integrity and size events out to the pipeline so we can use
// this new response body as an integrityEmitter for cacache
integrityStream.on('integrity', i => pipeline.emit('integrity', i))
integrityStream.on('size', s => pipeline.emit('size', s))
res = new fetch.Response(pipeline, res)
// set an explicit flag so we know if our response body will emit integrity and size
res.body.hasIntegrityEmitter = true
}
res.headers.set('x-fetch-attempts', attemptNum)
// do not retry POST requests, or requests with a streaming body
// do retry requests with a 408, 420, 429 or 500+ status in the response
const isStream = Minipass.isStream(req.body)
const isRetriable = req.method !== 'POST' &&
!isStream &&
([408, 420, 429].includes(res.status) || res.status >= 500)
if (isRetriable) {
if (typeof options.onRetry === 'function') {
options.onRetry(res)
}
return retryHandler(res)
}
return res
} catch (err) {
const code = (err.code === 'EPROMISERETRY')
? err.retried.code
: err.code
// err.retried will be the thing that was thrown from above
// if it's a response, we just got a bad status code and we
// can re-throw to allow the retry
const isRetryError = err.retried instanceof fetch.Response ||
(RETRY_ERRORS.includes(code) && RETRY_TYPES.includes(err.type))
if (req.method === 'POST' || isRetryError) {
throw err
}
if (typeof options.onRetry === 'function') {
options.onRetry(err)
}
return retryHandler(err)
}
}, options.retry).catch((err) => {
// don't reject for http errors, just return them
if (err.status >= 400 && err.type !== 'system') {
return err
}
throw err
})
}
module.exports = remoteFetch
@@ -0,0 +1,80 @@
{
"name": "make-fetch-happen",
"version": "11.0.2",
"description": "Opinionated, caching, retrying fetch client",
"main": "lib/index.js",
"files": [
"bin/",
"lib/"
],
"scripts": {
"test": "tap",
"posttest": "npm run lint",
"eslint": "eslint",
"lint": "eslint \"**/*.js\"",
"lintfix": "npm run lint -- --fix",
"postlint": "template-oss-check",
"snap": "tap",
"template-oss-apply": "template-oss-apply --force"
},
"repository": {
"type": "git",
"url": "https://github.com/npm/make-fetch-happen.git"
},
"keywords": [
"http",
"request",
"fetch",
"mean girls",
"caching",
"cache",
"subresource integrity"
],
"author": "GitHub Inc.",
"license": "ISC",
"dependencies": {
"agentkeepalive": "^4.2.1",
"cacache": "^17.0.0",
"http-cache-semantics": "^4.1.0",
"http-proxy-agent": "^5.0.0",
"https-proxy-agent": "^5.0.0",
"is-lambda": "^1.0.1",
"lru-cache": "^7.7.1",
"minipass": "^4.0.0",
"minipass-collect": "^1.0.2",
"minipass-fetch": "^3.0.0",
"minipass-flush": "^1.0.5",
"minipass-pipeline": "^1.2.4",
"negotiator": "^0.6.3",
"promise-retry": "^2.0.1",
"socks-proxy-agent": "^7.0.0",
"ssri": "^10.0.0"
},
"devDependencies": {
"@npmcli/eslint-config": "^4.0.0",
"@npmcli/template-oss": "4.10.0",
"mkdirp": "^1.0.4",
"nock": "^13.2.4",
"rimraf": "^3.0.2",
"safe-buffer": "^5.2.1",
"standard-version": "^9.3.2",
"tap": "^16.0.0"
},
"engines": {
"node": "^14.17.0 || ^16.13.0 || >=18.0.0"
},
"tap": {
"color": 1,
"files": "test/*.js",
"check-coverage": true,
"timeout": 60,
"nyc-arg": [
"--exclude",
"tap-snapshots/**"
]
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "4.10.0"
}
}
@@ -0,0 +1,28 @@
The MIT License (MIT)
Copyright (c) Isaac Z. Schlueter and Contributors
Copyright (c) 2016 David Frank
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
---
Note: This is a derivative work based on "node-fetch" by David Frank,
modified and distributed under the terms of the MIT license above.
https://github.com/bitinn/node-fetch
@@ -0,0 +1,29 @@
# minipass-fetch
An implementation of window.fetch in Node.js using Minipass streams
This is a fork (or more precisely, a reimplementation) of
[node-fetch](http://npm.im/node-fetch). All streams have been replaced
with [minipass streams](http://npm.im/minipass).
The goal of this module is to stay in sync with the API presented by
`node-fetch`, with the exception of the streaming interface provided.
## Why
Minipass streams are faster and more deterministic in their timing contract
than node-core streams, making them a better fit for many server-side use
cases.
## API
See [node-fetch](http://npm.im/node-fetch)
Differences from `node-fetch` (and, by extension, from the WhatWG Fetch
specification):
- Returns [minipass](http://npm.im/minipass) streams instead of node-core
streams.
- Supports the full set of [TLS Options that may be provided to
`https.request()`](https://nodejs.org/api/https.html#https_https_request_options_callback)
when making `https` requests.
@@ -0,0 +1,17 @@
'use strict'
class AbortError extends Error {
constructor (message) {
super(message)
this.code = 'FETCH_ABORTED'
this.type = 'aborted'
Error.captureStackTrace(this, this.constructor)
}
get name () {
return 'AbortError'
}
// don't allow name to be overridden, but don't throw either
set name (s) {}
}
module.exports = AbortError
@@ -0,0 +1,97 @@
'use strict'
const Minipass = require('minipass')
const TYPE = Symbol('type')
const BUFFER = Symbol('buffer')
class Blob {
constructor (blobParts, options) {
this[TYPE] = ''
const buffers = []
let size = 0
if (blobParts) {
const a = blobParts
const length = Number(a.length)
for (let i = 0; i < length; i++) {
const element = a[i]
const buffer = element instanceof Buffer ? element
: ArrayBuffer.isView(element)
? Buffer.from(element.buffer, element.byteOffset, element.byteLength)
: element instanceof ArrayBuffer ? Buffer.from(element)
: element instanceof Blob ? element[BUFFER]
: typeof element === 'string' ? Buffer.from(element)
: Buffer.from(String(element))
size += buffer.length
buffers.push(buffer)
}
}
this[BUFFER] = Buffer.concat(buffers, size)
const type = options && options.type !== undefined
&& String(options.type).toLowerCase()
if (type && !/[^\u0020-\u007E]/.test(type)) {
this[TYPE] = type
}
}
get size () {
return this[BUFFER].length
}
get type () {
return this[TYPE]
}
text () {
return Promise.resolve(this[BUFFER].toString())
}
arrayBuffer () {
const buf = this[BUFFER]
const off = buf.byteOffset
const len = buf.byteLength
const ab = buf.buffer.slice(off, off + len)
return Promise.resolve(ab)
}
stream () {
return new Minipass().end(this[BUFFER])
}
slice (start, end, type) {
const size = this.size
const relativeStart = start === undefined ? 0
: start < 0 ? Math.max(size + start, 0)
: Math.min(start, size)
const relativeEnd = end === undefined ? size
: end < 0 ? Math.max(size + end, 0)
: Math.min(end, size)
const span = Math.max(relativeEnd - relativeStart, 0)
const buffer = this[BUFFER]
const slicedBuffer = buffer.slice(
relativeStart,
relativeStart + span
)
const blob = new Blob([], { type })
blob[BUFFER] = slicedBuffer
return blob
}
get [Symbol.toStringTag] () {
return 'Blob'
}
static get BUFFER () {
return BUFFER
}
}
Object.defineProperties(Blob.prototype, {
size: { enumerable: true },
type: { enumerable: true },
})
module.exports = Blob
@@ -0,0 +1,350 @@
'use strict'
const Minipass = require('minipass')
const MinipassSized = require('minipass-sized')
const Blob = require('./blob.js')
const { BUFFER } = Blob
const FetchError = require('./fetch-error.js')
// optional dependency on 'encoding'
let convert
try {
convert = require('encoding').convert
} catch (e) {
// defer error until textConverted is called
}
const INTERNALS = Symbol('Body internals')
const CONSUME_BODY = Symbol('consumeBody')
class Body {
constructor (bodyArg, options = {}) {
const { size = 0, timeout = 0 } = options
const body = bodyArg === undefined || bodyArg === null ? null
: isURLSearchParams(bodyArg) ? Buffer.from(bodyArg.toString())
: isBlob(bodyArg) ? bodyArg
: Buffer.isBuffer(bodyArg) ? bodyArg
: Object.prototype.toString.call(bodyArg) === '[object ArrayBuffer]'
? Buffer.from(bodyArg)
: ArrayBuffer.isView(bodyArg)
? Buffer.from(bodyArg.buffer, bodyArg.byteOffset, bodyArg.byteLength)
: Minipass.isStream(bodyArg) ? bodyArg
: Buffer.from(String(bodyArg))
this[INTERNALS] = {
body,
disturbed: false,
error: null,
}
this.size = size
this.timeout = timeout
if (Minipass.isStream(body)) {
body.on('error', er => {
const error = er.name === 'AbortError' ? er
: new FetchError(`Invalid response while trying to fetch ${
this.url}: ${er.message}`, 'system', er)
this[INTERNALS].error = error
})
}
}
get body () {
return this[INTERNALS].body
}
get bodyUsed () {
return this[INTERNALS].disturbed
}
arrayBuffer () {
return this[CONSUME_BODY]().then(buf =>
buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.byteLength))
}
blob () {
const ct = this.headers && this.headers.get('content-type') || ''
return this[CONSUME_BODY]().then(buf => Object.assign(
new Blob([], { type: ct.toLowerCase() }),
{ [BUFFER]: buf }
))
}
async json () {
const buf = await this[CONSUME_BODY]()
try {
return JSON.parse(buf.toString())
} catch (er) {
throw new FetchError(
`invalid json response body at ${this.url} reason: ${er.message}`,
'invalid-json'
)
}
}
text () {
return this[CONSUME_BODY]().then(buf => buf.toString())
}
buffer () {
return this[CONSUME_BODY]()
}
textConverted () {
return this[CONSUME_BODY]().then(buf => convertBody(buf, this.headers))
}
[CONSUME_BODY] () {
if (this[INTERNALS].disturbed) {
return Promise.reject(new TypeError(`body used already for: ${
this.url}`))
}
this[INTERNALS].disturbed = true
if (this[INTERNALS].error) {
return Promise.reject(this[INTERNALS].error)
}
// body is null
if (this.body === null) {
return Promise.resolve(Buffer.alloc(0))
}
if (Buffer.isBuffer(this.body)) {
return Promise.resolve(this.body)
}
const upstream = isBlob(this.body) ? this.body.stream() : this.body
/* istanbul ignore if: should never happen */
if (!Minipass.isStream(upstream)) {
return Promise.resolve(Buffer.alloc(0))
}
const stream = this.size && upstream instanceof MinipassSized ? upstream
: !this.size && upstream instanceof Minipass &&
!(upstream instanceof MinipassSized) ? upstream
: this.size ? new MinipassSized({ size: this.size })
: new Minipass()
// allow timeout on slow response body, but only if the stream is still writable. this
// makes the timeout center on the socket stream from lib/index.js rather than the
// intermediary minipass stream we create to receive the data
const resTimeout = this.timeout && stream.writable ? setTimeout(() => {
stream.emit('error', new FetchError(
`Response timeout while trying to fetch ${
this.url} (over ${this.timeout}ms)`, 'body-timeout'))
}, this.timeout) : null
// do not keep the process open just for this timeout, even
// though we expect it'll get cleared eventually.
if (resTimeout && resTimeout.unref) {
resTimeout.unref()
}
// do the pipe in the promise, because the pipe() can send too much
// data through right away and upset the MP Sized object
return new Promise((resolve, reject) => {
// if the stream is some other kind of stream, then pipe through a MP
// so we can collect it more easily.
if (stream !== upstream) {
upstream.on('error', er => stream.emit('error', er))
upstream.pipe(stream)
}
resolve()
}).then(() => stream.concat()).then(buf => {
clearTimeout(resTimeout)
return buf
}).catch(er => {
clearTimeout(resTimeout)
// request was aborted, reject with this Error
if (er.name === 'AbortError' || er.name === 'FetchError') {
throw er
} else if (er.name === 'RangeError') {
throw new FetchError(`Could not create Buffer from response body for ${
this.url}: ${er.message}`, 'system', er)
} else {
// other errors, such as incorrect content-encoding or content-length
throw new FetchError(`Invalid response body while trying to fetch ${
this.url}: ${er.message}`, 'system', er)
}
})
}
static clone (instance) {
if (instance.bodyUsed) {
throw new Error('cannot clone body after it is used')
}
const body = instance.body
// check that body is a stream and not form-data object
// NB: can't clone the form-data object without having it as a dependency
if (Minipass.isStream(body) && typeof body.getBoundary !== 'function') {
// create a dedicated tee stream so that we don't lose data
// potentially sitting in the body stream's buffer by writing it
// immediately to p1 and not having it for p2.
const tee = new Minipass()
const p1 = new Minipass()
const p2 = new Minipass()
tee.on('error', er => {
p1.emit('error', er)
p2.emit('error', er)
})
body.on('error', er => tee.emit('error', er))
tee.pipe(p1)
tee.pipe(p2)
body.pipe(tee)
// set instance body to one fork, return the other
instance[INTERNALS].body = p1
return p2
} else {
return instance.body
}
}
static extractContentType (body) {
return body === null || body === undefined ? null
: typeof body === 'string' ? 'text/plain;charset=UTF-8'
: isURLSearchParams(body)
? 'application/x-www-form-urlencoded;charset=UTF-8'
: isBlob(body) ? body.type || null
: Buffer.isBuffer(body) ? null
: Object.prototype.toString.call(body) === '[object ArrayBuffer]' ? null
: ArrayBuffer.isView(body) ? null
: typeof body.getBoundary === 'function'
? `multipart/form-data;boundary=${body.getBoundary()}`
: Minipass.isStream(body) ? null
: 'text/plain;charset=UTF-8'
}
static getTotalBytes (instance) {
const { body } = instance
return (body === null || body === undefined) ? 0
: isBlob(body) ? body.size
: Buffer.isBuffer(body) ? body.length
: body && typeof body.getLengthSync === 'function' && (
// detect form data input from form-data module
body._lengthRetrievers &&
/* istanbul ignore next */ body._lengthRetrievers.length === 0 || // 1.x
body.hasKnownLength && body.hasKnownLength()) // 2.x
? body.getLengthSync()
: null
}
static writeToStream (dest, instance) {
const { body } = instance
if (body === null || body === undefined) {
dest.end()
} else if (Buffer.isBuffer(body) || typeof body === 'string') {
dest.end(body)
} else {
// body is stream or blob
const stream = isBlob(body) ? body.stream() : body
stream.on('error', er => dest.emit('error', er)).pipe(dest)
}
return dest
}
}
Object.defineProperties(Body.prototype, {
body: { enumerable: true },
bodyUsed: { enumerable: true },
arrayBuffer: { enumerable: true },
blob: { enumerable: true },
json: { enumerable: true },
text: { enumerable: true },
})
const isURLSearchParams = obj =>
// Duck-typing as a necessary condition.
(typeof obj !== 'object' ||
typeof obj.append !== 'function' ||
typeof obj.delete !== 'function' ||
typeof obj.get !== 'function' ||
typeof obj.getAll !== 'function' ||
typeof obj.has !== 'function' ||
typeof obj.set !== 'function') ? false
// Brand-checking and more duck-typing as optional condition.
: obj.constructor.name === 'URLSearchParams' ||
Object.prototype.toString.call(obj) === '[object URLSearchParams]' ||
typeof obj.sort === 'function'
const isBlob = obj =>
typeof obj === 'object' &&
typeof obj.arrayBuffer === 'function' &&
typeof obj.type === 'string' &&
typeof obj.stream === 'function' &&
typeof obj.constructor === 'function' &&
typeof obj.constructor.name === 'string' &&
/^(Blob|File)$/.test(obj.constructor.name) &&
/^(Blob|File)$/.test(obj[Symbol.toStringTag])
const convertBody = (buffer, headers) => {
/* istanbul ignore if */
if (typeof convert !== 'function') {
throw new Error('The package `encoding` must be installed to use the textConverted() function')
}
const ct = headers && headers.get('content-type')
let charset = 'utf-8'
let res
// header
if (ct) {
res = /charset=([^;]*)/i.exec(ct)
}
// no charset in content type, peek at response body for at most 1024 bytes
const str = buffer.slice(0, 1024).toString()
// html5
if (!res && str) {
res = /<meta.+?charset=(['"])(.+?)\1/i.exec(str)
}
// html4
if (!res && str) {
res = /<meta[\s]+?http-equiv=(['"])content-type\1[\s]+?content=(['"])(.+?)\2/i.exec(str)
if (!res) {
res = /<meta[\s]+?content=(['"])(.+?)\1[\s]+?http-equiv=(['"])content-type\3/i.exec(str)
if (res) {
res.pop()
} // drop last quote
}
if (res) {
res = /charset=(.*)/i.exec(res.pop())
}
}
// xml
if (!res && str) {
res = /<\?xml.+?encoding=(['"])(.+?)\1/i.exec(str)
}
// found charset
if (res) {
charset = res.pop()
// prevent decode issues when sites use incorrect encoding
// ref: https://hsivonen.fi/encoding-menu/
if (charset === 'gb2312' || charset === 'gbk') {
charset = 'gb18030'
}
}
// turn raw buffers into a single utf-8 buffer
return convert(
buffer,
'UTF-8',
charset
).toString()
}
module.exports = Body
@@ -0,0 +1,32 @@
'use strict'
class FetchError extends Error {
constructor (message, type, systemError) {
super(message)
this.code = 'FETCH_ERROR'
// pick up code, expected, path, ...
if (systemError) {
Object.assign(this, systemError)
}
this.errno = this.code
// override anything the system error might've clobbered
this.type = this.code === 'EBADSIZE' && this.found > this.expect
? 'max-size' : type
this.message = message
Error.captureStackTrace(this, this.constructor)
}
get name () {
return 'FetchError'
}
// don't allow name to be overwritten
set name (n) {}
get [Symbol.toStringTag] () {
return 'FetchError'
}
}
module.exports = FetchError
@@ -0,0 +1,267 @@
'use strict'
const invalidTokenRegex = /[^^_`a-zA-Z\-0-9!#$%&'*+.|~]/
const invalidHeaderCharRegex = /[^\t\x20-\x7e\x80-\xff]/
const validateName = name => {
name = `${name}`
if (invalidTokenRegex.test(name) || name === '') {
throw new TypeError(`${name} is not a legal HTTP header name`)
}
}
const validateValue = value => {
value = `${value}`
if (invalidHeaderCharRegex.test(value)) {
throw new TypeError(`${value} is not a legal HTTP header value`)
}
}
const find = (map, name) => {
name = name.toLowerCase()
for (const key in map) {
if (key.toLowerCase() === name) {
return key
}
}
return undefined
}
const MAP = Symbol('map')
class Headers {
constructor (init = undefined) {
this[MAP] = Object.create(null)
if (init instanceof Headers) {
const rawHeaders = init.raw()
const headerNames = Object.keys(rawHeaders)
for (const headerName of headerNames) {
for (const value of rawHeaders[headerName]) {
this.append(headerName, value)
}
}
return
}
// no-op
if (init === undefined || init === null) {
return
}
if (typeof init === 'object') {
const method = init[Symbol.iterator]
if (method !== null && method !== undefined) {
if (typeof method !== 'function') {
throw new TypeError('Header pairs must be iterable')
}
// sequence<sequence<ByteString>>
// Note: per spec we have to first exhaust the lists then process them
const pairs = []
for (const pair of init) {
if (typeof pair !== 'object' ||
typeof pair[Symbol.iterator] !== 'function') {
throw new TypeError('Each header pair must be iterable')
}
const arrPair = Array.from(pair)
if (arrPair.length !== 2) {
throw new TypeError('Each header pair must be a name/value tuple')
}
pairs.push(arrPair)
}
for (const pair of pairs) {
this.append(pair[0], pair[1])
}
} else {
// record<ByteString, ByteString>
for (const key of Object.keys(init)) {
this.append(key, init[key])
}
}
} else {
throw new TypeError('Provided initializer must be an object')
}
}
get (name) {
name = `${name}`
validateName(name)
const key = find(this[MAP], name)
if (key === undefined) {
return null
}
return this[MAP][key].join(', ')
}
forEach (callback, thisArg = undefined) {
let pairs = getHeaders(this)
for (let i = 0; i < pairs.length; i++) {
const [name, value] = pairs[i]
callback.call(thisArg, value, name, this)
// refresh in case the callback added more headers
pairs = getHeaders(this)
}
}
set (name, value) {
name = `${name}`
value = `${value}`
validateName(name)
validateValue(value)
const key = find(this[MAP], name)
this[MAP][key !== undefined ? key : name] = [value]
}
append (name, value) {
name = `${name}`
value = `${value}`
validateName(name)
validateValue(value)
const key = find(this[MAP], name)
if (key !== undefined) {
this[MAP][key].push(value)
} else {
this[MAP][name] = [value]
}
}
has (name) {
name = `${name}`
validateName(name)
return find(this[MAP], name) !== undefined
}
delete (name) {
name = `${name}`
validateName(name)
const key = find(this[MAP], name)
if (key !== undefined) {
delete this[MAP][key]
}
}
raw () {
return this[MAP]
}
keys () {
return new HeadersIterator(this, 'key')
}
values () {
return new HeadersIterator(this, 'value')
}
[Symbol.iterator] () {
return new HeadersIterator(this, 'key+value')
}
entries () {
return new HeadersIterator(this, 'key+value')
}
get [Symbol.toStringTag] () {
return 'Headers'
}
static exportNodeCompatibleHeaders (headers) {
const obj = Object.assign(Object.create(null), headers[MAP])
// http.request() only supports string as Host header. This hack makes
// specifying custom Host header possible.
const hostHeaderKey = find(headers[MAP], 'Host')
if (hostHeaderKey !== undefined) {
obj[hostHeaderKey] = obj[hostHeaderKey][0]
}
return obj
}
static createHeadersLenient (obj) {
const headers = new Headers()
for (const name of Object.keys(obj)) {
if (invalidTokenRegex.test(name)) {
continue
}
if (Array.isArray(obj[name])) {
for (const val of obj[name]) {
if (invalidHeaderCharRegex.test(val)) {
continue
}
if (headers[MAP][name] === undefined) {
headers[MAP][name] = [val]
} else {
headers[MAP][name].push(val)
}
}
} else if (!invalidHeaderCharRegex.test(obj[name])) {
headers[MAP][name] = [obj[name]]
}
}
return headers
}
}
Object.defineProperties(Headers.prototype, {
get: { enumerable: true },
forEach: { enumerable: true },
set: { enumerable: true },
append: { enumerable: true },
has: { enumerable: true },
delete: { enumerable: true },
keys: { enumerable: true },
values: { enumerable: true },
entries: { enumerable: true },
})
const getHeaders = (headers, kind = 'key+value') =>
Object.keys(headers[MAP]).sort().map(
kind === 'key' ? k => k.toLowerCase()
: kind === 'value' ? k => headers[MAP][k].join(', ')
: k => [k.toLowerCase(), headers[MAP][k].join(', ')]
)
const INTERNAL = Symbol('internal')
class HeadersIterator {
constructor (target, kind) {
this[INTERNAL] = {
target,
kind,
index: 0,
}
}
get [Symbol.toStringTag] () {
return 'HeadersIterator'
}
next () {
/* istanbul ignore if: should be impossible */
if (!this || Object.getPrototypeOf(this) !== HeadersIterator.prototype) {
throw new TypeError('Value of `this` is not a HeadersIterator')
}
const { target, kind, index } = this[INTERNAL]
const values = getHeaders(target, kind)
const len = values.length
if (index >= len) {
return {
value: undefined,
done: true,
}
}
this[INTERNAL].index++
return { value: values[index], done: false }
}
}
// manually extend because 'extends' requires a ctor
Object.setPrototypeOf(HeadersIterator.prototype,
Object.getPrototypeOf(Object.getPrototypeOf([][Symbol.iterator]())))
module.exports = Headers
@@ -0,0 +1,365 @@
'use strict'
const { URL } = require('url')
const http = require('http')
const https = require('https')
const zlib = require('minizlib')
const Minipass = require('minipass')
const Body = require('./body.js')
const { writeToStream, getTotalBytes } = Body
const Response = require('./response.js')
const Headers = require('./headers.js')
const { createHeadersLenient } = Headers
const Request = require('./request.js')
const { getNodeRequestOptions } = Request
const FetchError = require('./fetch-error.js')
const AbortError = require('./abort-error.js')
// XXX this should really be split up and unit-ized for easier testing
// and better DRY implementation of data/http request aborting
const fetch = async (url, opts) => {
if (/^data:/.test(url)) {
const request = new Request(url, opts)
// delay 1 promise tick so that the consumer can abort right away
return Promise.resolve().then(() => new Promise((resolve, reject) => {
let type, data
try {
const { pathname, search } = new URL(url)
const split = pathname.split(',')
if (split.length < 2) {
throw new Error('invalid data: URI')
}
const mime = split.shift()
const base64 = /;base64$/.test(mime)
type = base64 ? mime.slice(0, -1 * ';base64'.length) : mime
const rawData = decodeURIComponent(split.join(',') + search)
data = base64 ? Buffer.from(rawData, 'base64') : Buffer.from(rawData)
} catch (er) {
return reject(new FetchError(`[${request.method}] ${
request.url} invalid URL, ${er.message}`, 'system', er))
}
const { signal } = request
if (signal && signal.aborted) {
return reject(new AbortError('The user aborted a request.'))
}
const headers = { 'Content-Length': data.length }
if (type) {
headers['Content-Type'] = type
}
return resolve(new Response(data, { headers }))
}))
}
return new Promise((resolve, reject) => {
// build request object
const request = new Request(url, opts)
let options
try {
options = getNodeRequestOptions(request)
} catch (er) {
return reject(er)
}
const send = (options.protocol === 'https:' ? https : http).request
const { signal } = request
let response = null
const abort = () => {
const error = new AbortError('The user aborted a request.')
reject(error)
if (Minipass.isStream(request.body) &&
typeof request.body.destroy === 'function') {
request.body.destroy(error)
}
if (response && response.body) {
response.body.emit('error', error)
}
}
if (signal && signal.aborted) {
return abort()
}
const abortAndFinalize = () => {
abort()
finalize()
}
const finalize = () => {
req.abort()
if (signal) {
signal.removeEventListener('abort', abortAndFinalize)
}
clearTimeout(reqTimeout)
}
// send request
const req = send(options)
if (signal) {
signal.addEventListener('abort', abortAndFinalize)
}
let reqTimeout = null
if (request.timeout) {
req.once('socket', socket => {
reqTimeout = setTimeout(() => {
reject(new FetchError(`network timeout at: ${
request.url}`, 'request-timeout'))
finalize()
}, request.timeout)
})
}
req.on('error', er => {
// if a 'response' event is emitted before the 'error' event, then by the
// time this handler is run it's too late to reject the Promise for the
// response. instead, we forward the error event to the response stream
// so that the error will surface to the user when they try to consume
// the body. this is done as a side effect of aborting the request except
// for in windows, where we must forward the event manually, otherwise
// there is no longer a ref'd socket attached to the request and the
// stream never ends so the event loop runs out of work and the process
// exits without warning.
// coverage skipped here due to the difficulty in testing
// istanbul ignore next
if (req.res) {
req.res.emit('error', er)
}
reject(new FetchError(`request to ${request.url} failed, reason: ${
er.message}`, 'system', er))
finalize()
})
req.on('response', res => {
clearTimeout(reqTimeout)
const headers = createHeadersLenient(res.headers)
// HTTP fetch step 5
if (fetch.isRedirect(res.statusCode)) {
// HTTP fetch step 5.2
const location = headers.get('Location')
// HTTP fetch step 5.3
const locationURL = location === null ? null
: (new URL(location, request.url)).toString()
// HTTP fetch step 5.5
if (request.redirect === 'error') {
reject(new FetchError('uri requested responds with a redirect, ' +
`redirect mode is set to error: ${request.url}`, 'no-redirect'))
finalize()
return
} else if (request.redirect === 'manual') {
// node-fetch-specific step: make manual redirect a bit easier to
// use by setting the Location header value to the resolved URL.
if (locationURL !== null) {
// handle corrupted header
try {
headers.set('Location', locationURL)
} catch (err) {
/* istanbul ignore next: nodejs server prevent invalid
response headers, we can't test this through normal
request */
reject(err)
}
}
} else if (request.redirect === 'follow' && locationURL !== null) {
// HTTP-redirect fetch step 5
if (request.counter >= request.follow) {
reject(new FetchError(`maximum redirect reached at: ${
request.url}`, 'max-redirect'))
finalize()
return
}
// HTTP-redirect fetch step 9
if (res.statusCode !== 303 &&
request.body &&
getTotalBytes(request) === null) {
reject(new FetchError(
'Cannot follow redirect with body being a readable stream',
'unsupported-redirect'
))
finalize()
return
}
// Update host due to redirection
request.headers.set('host', (new URL(locationURL)).host)
// HTTP-redirect fetch step 6 (counter increment)
// Create a new Request object.
const requestOpts = {
headers: new Headers(request.headers),
follow: request.follow,
counter: request.counter + 1,
agent: request.agent,
compress: request.compress,
method: request.method,
body: request.body,
signal: request.signal,
timeout: request.timeout,
}
// if the redirect is to a new hostname, strip the authorization and cookie headers
const parsedOriginal = new URL(request.url)
const parsedRedirect = new URL(locationURL)
if (parsedOriginal.hostname !== parsedRedirect.hostname) {
requestOpts.headers.delete('authorization')
requestOpts.headers.delete('cookie')
}
// HTTP-redirect fetch step 11
if (res.statusCode === 303 || (
(res.statusCode === 301 || res.statusCode === 302) &&
request.method === 'POST'
)) {
requestOpts.method = 'GET'
requestOpts.body = undefined
requestOpts.headers.delete('content-length')
}
// HTTP-redirect fetch step 15
resolve(fetch(new Request(locationURL, requestOpts)))
finalize()
return
}
} // end if(isRedirect)
// prepare response
res.once('end', () =>
signal && signal.removeEventListener('abort', abortAndFinalize))
const body = new Minipass()
// if an error occurs, either on the response stream itself, on one of the
// decoder streams, or a response length timeout from the Body class, we
// forward the error through to our internal body stream. If we see an
// error event on that, we call finalize to abort the request and ensure
// we don't leave a socket believing a request is in flight.
// this is difficult to test, so lacks specific coverage.
body.on('error', finalize)
// exceedingly rare that the stream would have an error,
// but just in case we proxy it to the stream in use.
res.on('error', /* istanbul ignore next */ er => body.emit('error', er))
res.on('data', (chunk) => body.write(chunk))
res.on('end', () => body.end())
const responseOptions = {
url: request.url,
status: res.statusCode,
statusText: res.statusMessage,
headers: headers,
size: request.size,
timeout: request.timeout,
counter: request.counter,
trailer: new Promise(resolveTrailer =>
res.on('end', () => resolveTrailer(createHeadersLenient(res.trailers)))),
}
// HTTP-network fetch step 12.1.1.3
const codings = headers.get('Content-Encoding')
// HTTP-network fetch step 12.1.1.4: handle content codings
// in following scenarios we ignore compression support
// 1. compression support is disabled
// 2. HEAD request
// 3. no Content-Encoding header
// 4. no content response (204)
// 5. content not modified response (304)
if (!request.compress ||
request.method === 'HEAD' ||
codings === null ||
res.statusCode === 204 ||
res.statusCode === 304) {
response = new Response(body, responseOptions)
resolve(response)
return
}
// Be less strict when decoding compressed responses, since sometimes
// servers send slightly invalid responses that are still accepted
// by common browsers.
// Always using Z_SYNC_FLUSH is what cURL does.
const zlibOptions = {
flush: zlib.constants.Z_SYNC_FLUSH,
finishFlush: zlib.constants.Z_SYNC_FLUSH,
}
// for gzip
if (codings === 'gzip' || codings === 'x-gzip') {
const unzip = new zlib.Gunzip(zlibOptions)
response = new Response(
// exceedingly rare that the stream would have an error,
// but just in case we proxy it to the stream in use.
body.on('error', /* istanbul ignore next */ er => unzip.emit('error', er)).pipe(unzip),
responseOptions
)
resolve(response)
return
}
// for deflate
if (codings === 'deflate' || codings === 'x-deflate') {
// handle the infamous raw deflate response from old servers
// a hack for old IIS and Apache servers
const raw = res.pipe(new Minipass())
raw.once('data', chunk => {
// see http://stackoverflow.com/questions/37519828
const decoder = (chunk[0] & 0x0F) === 0x08
? new zlib.Inflate()
: new zlib.InflateRaw()
// exceedingly rare that the stream would have an error,
// but just in case we proxy it to the stream in use.
body.on('error', /* istanbul ignore next */ er => decoder.emit('error', er)).pipe(decoder)
response = new Response(decoder, responseOptions)
resolve(response)
})
return
}
// for br
if (codings === 'br') {
// ignoring coverage so tests don't have to fake support (or lack of) for brotli
// istanbul ignore next
try {
var decoder = new zlib.BrotliDecompress()
} catch (err) {
reject(err)
finalize()
return
}
// exceedingly rare that the stream would have an error,
// but just in case we proxy it to the stream in use.
body.on('error', /* istanbul ignore next */ er => decoder.emit('error', er)).pipe(decoder)
response = new Response(decoder, responseOptions)
resolve(response)
return
}
// otherwise, use response as-is
response = new Response(body, responseOptions)
resolve(response)
})
writeToStream(req, request)
})
}
module.exports = fetch
fetch.isRedirect = code =>
code === 301 ||
code === 302 ||
code === 303 ||
code === 307 ||
code === 308
fetch.Headers = Headers
fetch.Request = Request
fetch.Response = Response
fetch.FetchError = FetchError
fetch.AbortError = AbortError
@@ -0,0 +1,281 @@
'use strict'
const { URL } = require('url')
const Minipass = require('minipass')
const Headers = require('./headers.js')
const { exportNodeCompatibleHeaders } = Headers
const Body = require('./body.js')
const { clone, extractContentType, getTotalBytes } = Body
const version = require('../package.json').version
const defaultUserAgent =
`minipass-fetch/${version} (+https://github.com/isaacs/minipass-fetch)`
const INTERNALS = Symbol('Request internals')
const isRequest = input =>
typeof input === 'object' && typeof input[INTERNALS] === 'object'
const isAbortSignal = signal => {
const proto = (
signal
&& typeof signal === 'object'
&& Object.getPrototypeOf(signal)
)
return !!(proto && proto.constructor.name === 'AbortSignal')
}
class Request extends Body {
constructor (input, init = {}) {
const parsedURL = isRequest(input) ? new URL(input.url)
: input && input.href ? new URL(input.href)
: new URL(`${input}`)
if (isRequest(input)) {
init = { ...input[INTERNALS], ...init }
} else if (!input || typeof input === 'string') {
input = {}
}
const method = (init.method || input.method || 'GET').toUpperCase()
const isGETHEAD = method === 'GET' || method === 'HEAD'
if ((init.body !== null && init.body !== undefined ||
isRequest(input) && input.body !== null) && isGETHEAD) {
throw new TypeError('Request with GET/HEAD method cannot have body')
}
const inputBody = init.body !== null && init.body !== undefined ? init.body
: isRequest(input) && input.body !== null ? clone(input)
: null
super(inputBody, {
timeout: init.timeout || input.timeout || 0,
size: init.size || input.size || 0,
})
const headers = new Headers(init.headers || input.headers || {})
if (inputBody !== null && inputBody !== undefined &&
!headers.has('Content-Type')) {
const contentType = extractContentType(inputBody)
if (contentType) {
headers.append('Content-Type', contentType)
}
}
const signal = 'signal' in init ? init.signal
: null
if (signal !== null && signal !== undefined && !isAbortSignal(signal)) {
throw new TypeError('Expected signal must be an instanceof AbortSignal')
}
// TLS specific options that are handled by node
const {
ca,
cert,
ciphers,
clientCertEngine,
crl,
dhparam,
ecdhCurve,
family,
honorCipherOrder,
key,
passphrase,
pfx,
rejectUnauthorized = process.env.NODE_TLS_REJECT_UNAUTHORIZED !== '0',
secureOptions,
secureProtocol,
servername,
sessionIdContext,
} = init
this[INTERNALS] = {
method,
redirect: init.redirect || input.redirect || 'follow',
headers,
parsedURL,
signal,
ca,
cert,
ciphers,
clientCertEngine,
crl,
dhparam,
ecdhCurve,
family,
honorCipherOrder,
key,
passphrase,
pfx,
rejectUnauthorized,
secureOptions,
secureProtocol,
servername,
sessionIdContext,
}
// node-fetch-only options
this.follow = init.follow !== undefined ? init.follow
: input.follow !== undefined ? input.follow
: 20
this.compress = init.compress !== undefined ? init.compress
: input.compress !== undefined ? input.compress
: true
this.counter = init.counter || input.counter || 0
this.agent = init.agent || input.agent
}
get method () {
return this[INTERNALS].method
}
get url () {
return this[INTERNALS].parsedURL.toString()
}
get headers () {
return this[INTERNALS].headers
}
get redirect () {
return this[INTERNALS].redirect
}
get signal () {
return this[INTERNALS].signal
}
clone () {
return new Request(this)
}
get [Symbol.toStringTag] () {
return 'Request'
}
static getNodeRequestOptions (request) {
const parsedURL = request[INTERNALS].parsedURL
const headers = new Headers(request[INTERNALS].headers)
// fetch step 1.3
if (!headers.has('Accept')) {
headers.set('Accept', '*/*')
}
// Basic fetch
if (!/^https?:$/.test(parsedURL.protocol)) {
throw new TypeError('Only HTTP(S) protocols are supported')
}
if (request.signal &&
Minipass.isStream(request.body) &&
typeof request.body.destroy !== 'function') {
throw new Error(
'Cancellation of streamed requests with AbortSignal is not supported')
}
// HTTP-network-or-cache fetch steps 2.4-2.7
const contentLengthValue =
(request.body === null || request.body === undefined) &&
/^(POST|PUT)$/i.test(request.method) ? '0'
: request.body !== null && request.body !== undefined
? getTotalBytes(request)
: null
if (contentLengthValue) {
headers.set('Content-Length', contentLengthValue + '')
}
// HTTP-network-or-cache fetch step 2.11
if (!headers.has('User-Agent')) {
headers.set('User-Agent', defaultUserAgent)
}
// HTTP-network-or-cache fetch step 2.15
if (request.compress && !headers.has('Accept-Encoding')) {
headers.set('Accept-Encoding', 'gzip,deflate')
}
const agent = typeof request.agent === 'function'
? request.agent(parsedURL)
: request.agent
if (!headers.has('Connection') && !agent) {
headers.set('Connection', 'close')
}
// TLS specific options that are handled by node
const {
ca,
cert,
ciphers,
clientCertEngine,
crl,
dhparam,
ecdhCurve,
family,
honorCipherOrder,
key,
passphrase,
pfx,
rejectUnauthorized,
secureOptions,
secureProtocol,
servername,
sessionIdContext,
} = request[INTERNALS]
// HTTP-network fetch step 4.2
// chunked encoding is handled by Node.js
// we cannot spread parsedURL directly, so we have to read each property one-by-one
// and map them to the equivalent https?.request() method options
const urlProps = {
auth: parsedURL.username || parsedURL.password
? `${parsedURL.username}:${parsedURL.password}`
: '',
host: parsedURL.host,
hostname: parsedURL.hostname,
path: `${parsedURL.pathname}${parsedURL.search}`,
port: parsedURL.port,
protocol: parsedURL.protocol,
}
return {
...urlProps,
method: request.method,
headers: exportNodeCompatibleHeaders(headers),
agent,
ca,
cert,
ciphers,
clientCertEngine,
crl,
dhparam,
ecdhCurve,
family,
honorCipherOrder,
key,
passphrase,
pfx,
rejectUnauthorized,
secureOptions,
secureProtocol,
servername,
sessionIdContext,
}
}
}
module.exports = Request
Object.defineProperties(Request.prototype, {
method: { enumerable: true },
url: { enumerable: true },
headers: { enumerable: true },
redirect: { enumerable: true },
clone: { enumerable: true },
signal: { enumerable: true },
})
@@ -0,0 +1,90 @@
'use strict'
const http = require('http')
const { STATUS_CODES } = http
const Headers = require('./headers.js')
const Body = require('./body.js')
const { clone, extractContentType } = Body
const INTERNALS = Symbol('Response internals')
class Response extends Body {
constructor (body = null, opts = {}) {
super(body, opts)
const status = opts.status || 200
const headers = new Headers(opts.headers)
if (body !== null && body !== undefined && !headers.has('Content-Type')) {
const contentType = extractContentType(body)
if (contentType) {
headers.append('Content-Type', contentType)
}
}
this[INTERNALS] = {
url: opts.url,
status,
statusText: opts.statusText || STATUS_CODES[status],
headers,
counter: opts.counter,
trailer: Promise.resolve(opts.trailer || new Headers()),
}
}
get trailer () {
return this[INTERNALS].trailer
}
get url () {
return this[INTERNALS].url || ''
}
get status () {
return this[INTERNALS].status
}
get ok () {
return this[INTERNALS].status >= 200 && this[INTERNALS].status < 300
}
get redirected () {
return this[INTERNALS].counter > 0
}
get statusText () {
return this[INTERNALS].statusText
}
get headers () {
return this[INTERNALS].headers
}
clone () {
return new Response(clone(this), {
url: this.url,
status: this.status,
statusText: this.statusText,
headers: this.headers,
ok: this.ok,
redirected: this.redirected,
trailer: this.trailer,
})
}
get [Symbol.toStringTag] () {
return 'Response'
}
}
module.exports = Response
Object.defineProperties(Response.prototype, {
url: { enumerable: true },
status: { enumerable: true },
ok: { enumerable: true },
redirected: { enumerable: true },
statusText: { enumerable: true },
headers: { enumerable: true },
clone: { enumerable: true },
})
@@ -0,0 +1,68 @@
{
"name": "minipass-fetch",
"version": "3.0.1",
"description": "An implementation of window.fetch in Node.js using Minipass streams",
"license": "MIT",
"main": "lib/index.js",
"scripts": {
"test:tls-fixtures": "./test/fixtures/tls/setup.sh",
"test": "tap",
"snap": "tap",
"lint": "eslint \"**/*.js\"",
"postlint": "template-oss-check",
"lintfix": "npm run lint -- --fix",
"posttest": "npm run lint",
"template-oss-apply": "template-oss-apply --force"
},
"tap": {
"coverage-map": "map.js",
"check-coverage": true,
"nyc-arg": [
"--exclude",
"tap-snapshots/**"
]
},
"devDependencies": {
"@npmcli/eslint-config": "^4.0.0",
"@npmcli/template-oss": "4.10.0",
"@ungap/url-search-params": "^0.2.2",
"abort-controller": "^3.0.0",
"abortcontroller-polyfill": "~1.7.3",
"encoding": "^0.1.13",
"form-data": "^4.0.0",
"nock": "^13.2.4",
"parted": "^0.1.1",
"string-to-arraybuffer": "^1.0.2",
"tap": "^16.0.0"
},
"dependencies": {
"minipass": "^4.0.0",
"minipass-sized": "^1.0.3",
"minizlib": "^2.1.2"
},
"optionalDependencies": {
"encoding": "^0.1.13"
},
"repository": {
"type": "git",
"url": "https://github.com/npm/minipass-fetch.git"
},
"keywords": [
"fetch",
"minipass",
"node-fetch",
"window.fetch"
],
"files": [
"bin/",
"lib/"
],
"engines": {
"node": "^14.17.0 || ^16.13.0 || >=18.0.0"
},
"author": "GitHub Inc.",
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "4.10.0"
}
}
@@ -0,0 +1,15 @@
The ISC License
Copyright (c) 2017-2022 npm, Inc., Isaac Z. Schlueter, and Contributors
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
@@ -0,0 +1,728 @@
# minipass
A _very_ minimal implementation of a [PassThrough
stream](https://nodejs.org/api/stream.html#stream_class_stream_passthrough)
[It's very
fast](https://docs.google.com/spreadsheets/d/1oObKSrVwLX_7Ut4Z6g3fZW-AX1j1-k6w-cDsrkaSbHM/edit#gid=0)
for objects, strings, and buffers.
Supports `pipe()`ing (including multi-`pipe()` and backpressure transmission),
buffering data until either a `data` event handler or `pipe()` is added (so
you don't lose the first chunk), and most other cases where PassThrough is
a good idea.
There is a `read()` method, but it's much more efficient to consume data
from this stream via `'data'` events or by calling `pipe()` into some other
stream. Calling `read()` requires the buffer to be flattened in some
cases, which requires copying memory.
If you set `objectMode: true` in the options, then whatever is written will
be emitted. Otherwise, it'll do a minimal amount of Buffer copying to
ensure proper Streams semantics when `read(n)` is called.
`objectMode` can also be set by doing `stream.objectMode = true`, or by
writing any non-string/non-buffer data. `objectMode` cannot be set to
false once it is set.
This is not a `through` or `through2` stream. It doesn't transform the
data, it just passes it right through. If you want to transform the data,
extend the class, and override the `write()` method. Once you're done
transforming the data however you want, call `super.write()` with the
transform output.
For some examples of streams that extend Minipass in various ways, check
out:
- [minizlib](http://npm.im/minizlib)
- [fs-minipass](http://npm.im/fs-minipass)
- [tar](http://npm.im/tar)
- [minipass-collect](http://npm.im/minipass-collect)
- [minipass-flush](http://npm.im/minipass-flush)
- [minipass-pipeline](http://npm.im/minipass-pipeline)
- [tap](http://npm.im/tap)
- [tap-parser](http://npm.im/tap-parser)
- [treport](http://npm.im/treport)
- [minipass-fetch](http://npm.im/minipass-fetch)
- [pacote](http://npm.im/pacote)
- [make-fetch-happen](http://npm.im/make-fetch-happen)
- [cacache](http://npm.im/cacache)
- [ssri](http://npm.im/ssri)
- [npm-registry-fetch](http://npm.im/npm-registry-fetch)
- [minipass-json-stream](http://npm.im/minipass-json-stream)
- [minipass-sized](http://npm.im/minipass-sized)
## Differences from Node.js Streams
There are several things that make Minipass streams different from (and in
some ways superior to) Node.js core streams.
Please read these caveats if you are familiar with node-core streams and
intend to use Minipass streams in your programs.
You can avoid most of these differences entirely (for a very
small performance penalty) by setting `{async: true}` in the
constructor options.
### Timing
Minipass streams are designed to support synchronous use-cases. Thus, data
is emitted as soon as it is available, always. It is buffered until read,
but no longer. Another way to look at it is that Minipass streams are
exactly as synchronous as the logic that writes into them.
This can be surprising if your code relies on `PassThrough.write()` always
providing data on the next tick rather than the current one, or being able
to call `resume()` and not have the entire buffer disappear immediately.
However, without this synchronicity guarantee, there would be no way for
Minipass to achieve the speeds it does, or support the synchronous use
cases that it does. Simply put, waiting takes time.
This non-deferring approach makes Minipass streams much easier to reason
about, especially in the context of Promises and other flow-control
mechanisms.
Example:
```js
const Minipass = require('minipass')
const stream = new Minipass({ async: true })
stream.on('data', () => console.log('data event'))
console.log('before write')
stream.write('hello')
console.log('after write')
// output:
// before write
// data event
// after write
```
### Exception: Async Opt-In
If you wish to have a Minipass stream with behavior that more
closely mimics Node.js core streams, you can set the stream in
async mode either by setting `async: true` in the constructor
options, or by setting `stream.async = true` later on.
```js
const Minipass = require('minipass')
const asyncStream = new Minipass({ async: true })
asyncStream.on('data', () => console.log('data event'))
console.log('before write')
asyncStream.write('hello')
console.log('after write')
// output:
// before write
// after write
// data event <-- this is deferred until the next tick
```
Switching _out_ of async mode is unsafe, as it could cause data
corruption, and so is not enabled. Example:
```js
const Minipass = require('minipass')
const stream = new Minipass({ encoding: 'utf8' })
stream.on('data', chunk => console.log(chunk))
stream.async = true
console.log('before writes')
stream.write('hello')
setStreamSyncAgainSomehow(stream) // <-- this doesn't actually exist!
stream.write('world')
console.log('after writes')
// hypothetical output would be:
// before writes
// world
// after writes
// hello
// NOT GOOD!
```
To avoid this problem, once set into async mode, any attempt to
make the stream sync again will be ignored.
```js
const Minipass = require('minipass')
const stream = new Minipass({ encoding: 'utf8' })
stream.on('data', chunk => console.log(chunk))
stream.async = true
console.log('before writes')
stream.write('hello')
stream.async = false // <-- no-op, stream already async
stream.write('world')
console.log('after writes')
// actual output:
// before writes
// after writes
// hello
// world
```
### No High/Low Water Marks
Node.js core streams will optimistically fill up a buffer, returning `true`
on all writes until the limit is hit, even if the data has nowhere to go.
Then, they will not attempt to draw more data in until the buffer size dips
below a minimum value.
Minipass streams are much simpler. The `write()` method will return `true`
if the data has somewhere to go (which is to say, given the timing
guarantees, that the data is already there by the time `write()` returns).
If the data has nowhere to go, then `write()` returns false, and the data
sits in a buffer, to be drained out immediately as soon as anyone consumes
it.
Since nothing is ever buffered unnecessarily, there is much less
copying data, and less bookkeeping about buffer capacity levels.
### Hazards of Buffering (or: Why Minipass Is So Fast)
Since data written to a Minipass stream is immediately written all the way
through the pipeline, and `write()` always returns true/false based on
whether the data was fully flushed, backpressure is communicated
immediately to the upstream caller. This minimizes buffering.
Consider this case:
```js
const {PassThrough} = require('stream')
const p1 = new PassThrough({ highWaterMark: 1024 })
const p2 = new PassThrough({ highWaterMark: 1024 })
const p3 = new PassThrough({ highWaterMark: 1024 })
const p4 = new PassThrough({ highWaterMark: 1024 })
p1.pipe(p2).pipe(p3).pipe(p4)
p4.on('data', () => console.log('made it through'))
// this returns false and buffers, then writes to p2 on next tick (1)
// p2 returns false and buffers, pausing p1, then writes to p3 on next tick (2)
// p3 returns false and buffers, pausing p2, then writes to p4 on next tick (3)
// p4 returns false and buffers, pausing p3, then emits 'data' and 'drain'
// on next tick (4)
// p3 sees p4's 'drain' event, and calls resume(), emitting 'resume' and
// 'drain' on next tick (5)
// p2 sees p3's 'drain', calls resume(), emits 'resume' and 'drain' on next tick (6)
// p1 sees p2's 'drain', calls resume(), emits 'resume' and 'drain' on next
// tick (7)
p1.write(Buffer.alloc(2048)) // returns false
```
Along the way, the data was buffered and deferred at each stage, and
multiple event deferrals happened, for an unblocked pipeline where it was
perfectly safe to write all the way through!
Furthermore, setting a `highWaterMark` of `1024` might lead someone reading
the code to think an advisory maximum of 1KiB is being set for the
pipeline. However, the actual advisory buffering level is the _sum_ of
`highWaterMark` values, since each one has its own bucket.
Consider the Minipass case:
```js
const m1 = new Minipass()
const m2 = new Minipass()
const m3 = new Minipass()
const m4 = new Minipass()
m1.pipe(m2).pipe(m3).pipe(m4)
m4.on('data', () => console.log('made it through'))
// m1 is flowing, so it writes the data to m2 immediately
// m2 is flowing, so it writes the data to m3 immediately
// m3 is flowing, so it writes the data to m4 immediately
// m4 is flowing, so it fires the 'data' event immediately, returns true
// m4's write returned true, so m3 is still flowing, returns true
// m3's write returned true, so m2 is still flowing, returns true
// m2's write returned true, so m1 is still flowing, returns true
// No event deferrals or buffering along the way!
m1.write(Buffer.alloc(2048)) // returns true
```
It is extremely unlikely that you _don't_ want to buffer any data written,
or _ever_ buffer data that can be flushed all the way through. Neither
node-core streams nor Minipass ever fail to buffer written data, but
node-core streams do a lot of unnecessary buffering and pausing.
As always, the faster implementation is the one that does less stuff and
waits less time to do it.
### Immediately emit `end` for empty streams (when not paused)
If a stream is not paused, and `end()` is called before writing any data
into it, then it will emit `end` immediately.
If you have logic that occurs on the `end` event which you don't want to
potentially happen immediately (for example, closing file descriptors,
moving on to the next entry in an archive parse stream, etc.) then be sure
to call `stream.pause()` on creation, and then `stream.resume()` once you
are ready to respond to the `end` event.
However, this is _usually_ not a problem because:
### Emit `end` When Asked
One hazard of immediately emitting `'end'` is that you may not yet have had
a chance to add a listener. In order to avoid this hazard, Minipass
streams safely re-emit the `'end'` event if a new listener is added after
`'end'` has been emitted.
Ie, if you do `stream.on('end', someFunction)`, and the stream has already
emitted `end`, then it will call the handler right away. (You can think of
this somewhat like attaching a new `.then(fn)` to a previously-resolved
Promise.)
To prevent calling handlers multiple times who would not expect multiple
ends to occur, all listeners are removed from the `'end'` event whenever it
is emitted.
### Emit `error` When Asked
The most recent error object passed to the `'error'` event is
stored on the stream. If a new `'error'` event handler is added,
and an error was previously emitted, then the event handler will
be called immediately (or on `process.nextTick` in the case of
async streams).
This makes it much more difficult to end up trying to interact
with a broken stream, if the error handler is added after an
error was previously emitted.
### Impact of "immediate flow" on Tee-streams
A "tee stream" is a stream piping to multiple destinations:
```js
const tee = new Minipass()
t.pipe(dest1)
t.pipe(dest2)
t.write('foo') // goes to both destinations
```
Since Minipass streams _immediately_ process any pending data through the
pipeline when a new pipe destination is added, this can have surprising
effects, especially when a stream comes in from some other function and may
or may not have data in its buffer.
```js
// WARNING! WILL LOSE DATA!
const src = new Minipass()
src.write('foo')
src.pipe(dest1) // 'foo' chunk flows to dest1 immediately, and is gone
src.pipe(dest2) // gets nothing!
```
One solution is to create a dedicated tee-stream junction that pipes to
both locations, and then pipe to _that_ instead.
```js
// Safe example: tee to both places
const src = new Minipass()
src.write('foo')
const tee = new Minipass()
tee.pipe(dest1)
tee.pipe(dest2)
src.pipe(tee) // tee gets 'foo', pipes to both locations
```
The same caveat applies to `on('data')` event listeners. The first one
added will _immediately_ receive all of the data, leaving nothing for the
second:
```js
// WARNING! WILL LOSE DATA!
const src = new Minipass()
src.write('foo')
src.on('data', handler1) // receives 'foo' right away
src.on('data', handler2) // nothing to see here!
```
Using a dedicated tee-stream can be used in this case as well:
```js
// Safe example: tee to both data handlers
const src = new Minipass()
src.write('foo')
const tee = new Minipass()
tee.on('data', handler1)
tee.on('data', handler2)
src.pipe(tee)
```
All of the hazards in this section are avoided by setting `{
async: true }` in the Minipass constructor, or by setting
`stream.async = true` afterwards. Note that this does add some
overhead, so should only be done in cases where you are willing
to lose a bit of performance in order to avoid having to refactor
program logic.
## USAGE
It's a stream! Use it like a stream and it'll most likely do what you
want.
```js
const Minipass = require('minipass')
const mp = new Minipass(options) // optional: { encoding, objectMode }
mp.write('foo')
mp.pipe(someOtherStream)
mp.end('bar')
```
### OPTIONS
* `encoding` How would you like the data coming _out_ of the stream to be
encoded? Accepts any values that can be passed to `Buffer.toString()`.
* `objectMode` Emit data exactly as it comes in. This will be flipped on
by default if you write() something other than a string or Buffer at any
point. Setting `objectMode: true` will prevent setting any encoding
value.
* `async` Defaults to `false`. Set to `true` to defer data
emission until next tick. This reduces performance slightly,
but makes Minipass streams use timing behavior closer to Node
core streams. See [Timing](#timing) for more details.
### API
Implements the user-facing portions of Node.js's `Readable` and `Writable`
streams.
### Methods
* `write(chunk, [encoding], [callback])` - Put data in. (Note that, in the
base Minipass class, the same data will come out.) Returns `false` if
the stream will buffer the next write, or true if it's still in "flowing"
mode.
* `end([chunk, [encoding]], [callback])` - Signal that you have no more
data to write. This will queue an `end` event to be fired when all the
data has been consumed.
* `setEncoding(encoding)` - Set the encoding for data coming of the stream.
This can only be done once.
* `pause()` - No more data for a while, please. This also prevents `end`
from being emitted for empty streams until the stream is resumed.
* `resume()` - Resume the stream. If there's data in the buffer, it is all
discarded. Any buffered events are immediately emitted.
* `pipe(dest)` - Send all output to the stream provided. When
data is emitted, it is immediately written to any and all pipe
destinations. (Or written on next tick in `async` mode.)
* `unpipe(dest)` - Stop piping to the destination stream. This
is immediate, meaning that any asynchronously queued data will
_not_ make it to the destination when running in `async` mode.
* `options.end` - Boolean, end the destination stream when
the source stream ends. Default `true`.
* `options.proxyErrors` - Boolean, proxy `error` events from
the source stream to the destination stream. Note that
errors are _not_ proxied after the pipeline terminates,
either due to the source emitting `'end'` or manually
unpiping with `src.unpipe(dest)`. Default `false`.
* `on(ev, fn)`, `emit(ev, fn)` - Minipass streams are EventEmitters. Some
events are given special treatment, however. (See below under "events".)
* `promise()` - Returns a Promise that resolves when the stream emits
`end`, or rejects if the stream emits `error`.
* `collect()` - Return a Promise that resolves on `end` with an array
containing each chunk of data that was emitted, or rejects if the stream
emits `error`. Note that this consumes the stream data.
* `concat()` - Same as `collect()`, but concatenates the data into a single
Buffer object. Will reject the returned promise if the stream is in
objectMode, or if it goes into objectMode by the end of the data.
* `read(n)` - Consume `n` bytes of data out of the buffer. If `n` is not
provided, then consume all of it. If `n` bytes are not available, then
it returns null. **Note** consuming streams in this way is less
efficient, and can lead to unnecessary Buffer copying.
* `destroy([er])` - Destroy the stream. If an error is provided, then an
`'error'` event is emitted. If the stream has a `close()` method, and
has not emitted a `'close'` event yet, then `stream.close()` will be
called. Any Promises returned by `.promise()`, `.collect()` or
`.concat()` will be rejected. After being destroyed, writing to the
stream will emit an error. No more data will be emitted if the stream is
destroyed, even if it was previously buffered.
### Properties
* `bufferLength` Read-only. Total number of bytes buffered, or in the case
of objectMode, the total number of objects.
* `encoding` The encoding that has been set. (Setting this is equivalent
to calling `setEncoding(enc)` and has the same prohibition against
setting multiple times.)
* `flowing` Read-only. Boolean indicating whether a chunk written to the
stream will be immediately emitted.
* `emittedEnd` Read-only. Boolean indicating whether the end-ish events
(ie, `end`, `prefinish`, `finish`) have been emitted. Note that
listening on any end-ish event will immediateyl re-emit it if it has
already been emitted.
* `writable` Whether the stream is writable. Default `true`. Set to
`false` when `end()`
* `readable` Whether the stream is readable. Default `true`.
* `buffer` A [yallist](http://npm.im/yallist) linked list of chunks written
to the stream that have not yet been emitted. (It's probably a bad idea
to mess with this.)
* `pipes` A [yallist](http://npm.im/yallist) linked list of streams that
this stream is piping into. (It's probably a bad idea to mess with
this.)
* `destroyed` A getter that indicates whether the stream was destroyed.
* `paused` True if the stream has been explicitly paused, otherwise false.
* `objectMode` Indicates whether the stream is in `objectMode`. Once set
to `true`, it cannot be set to `false`.
### Events
* `data` Emitted when there's data to read. Argument is the data to read.
This is never emitted while not flowing. If a listener is attached, that
will resume the stream.
* `end` Emitted when there's no more data to read. This will be emitted
immediately for empty streams when `end()` is called. If a listener is
attached, and `end` was already emitted, then it will be emitted again.
All listeners are removed when `end` is emitted.
* `prefinish` An end-ish event that follows the same logic as `end` and is
emitted in the same conditions where `end` is emitted. Emitted after
`'end'`.
* `finish` An end-ish event that follows the same logic as `end` and is
emitted in the same conditions where `end` is emitted. Emitted after
`'prefinish'`.
* `close` An indication that an underlying resource has been released.
Minipass does not emit this event, but will defer it until after `end`
has been emitted, since it throws off some stream libraries otherwise.
* `drain` Emitted when the internal buffer empties, and it is again
suitable to `write()` into the stream.
* `readable` Emitted when data is buffered and ready to be read by a
consumer.
* `resume` Emitted when stream changes state from buffering to flowing
mode. (Ie, when `resume` is called, `pipe` is called, or a `data` event
listener is added.)
### Static Methods
* `Minipass.isStream(stream)` Returns `true` if the argument is a stream,
and false otherwise. To be considered a stream, the object must be
either an instance of Minipass, or an EventEmitter that has either a
`pipe()` method, or both `write()` and `end()` methods. (Pretty much any
stream in node-land will return `true` for this.)
## EXAMPLES
Here are some examples of things you can do with Minipass streams.
### simple "are you done yet" promise
```js
mp.promise().then(() => {
// stream is finished
}, er => {
// stream emitted an error
})
```
### collecting
```js
mp.collect().then(all => {
// all is an array of all the data emitted
// encoding is supported in this case, so
// so the result will be a collection of strings if
// an encoding is specified, or buffers/objects if not.
//
// In an async function, you may do
// const data = await stream.collect()
})
```
### collecting into a single blob
This is a bit slower because it concatenates the data into one chunk for
you, but if you're going to do it yourself anyway, it's convenient this
way:
```js
mp.concat().then(onebigchunk => {
// onebigchunk is a string if the stream
// had an encoding set, or a buffer otherwise.
})
```
### iteration
You can iterate over streams synchronously or asynchronously in platforms
that support it.
Synchronous iteration will end when the currently available data is
consumed, even if the `end` event has not been reached. In string and
buffer mode, the data is concatenated, so unless multiple writes are
occurring in the same tick as the `read()`, sync iteration loops will
generally only have a single iteration.
To consume chunks in this way exactly as they have been written, with no
flattening, create the stream with the `{ objectMode: true }` option.
```js
const mp = new Minipass({ objectMode: true })
mp.write('a')
mp.write('b')
for (let letter of mp) {
console.log(letter) // a, b
}
mp.write('c')
mp.write('d')
for (let letter of mp) {
console.log(letter) // c, d
}
mp.write('e')
mp.end()
for (let letter of mp) {
console.log(letter) // e
}
for (let letter of mp) {
console.log(letter) // nothing
}
```
Asynchronous iteration will continue until the end event is reached,
consuming all of the data.
```js
const mp = new Minipass({ encoding: 'utf8' })
// some source of some data
let i = 5
const inter = setInterval(() => {
if (i-- > 0)
mp.write(Buffer.from('foo\n', 'utf8'))
else {
mp.end()
clearInterval(inter)
}
}, 100)
// consume the data with asynchronous iteration
async function consume () {
for await (let chunk of mp) {
console.log(chunk)
}
return 'ok'
}
consume().then(res => console.log(res))
// logs `foo\n` 5 times, and then `ok`
```
### subclass that `console.log()`s everything written into it
```js
class Logger extends Minipass {
write (chunk, encoding, callback) {
console.log('WRITE', chunk, encoding)
return super.write(chunk, encoding, callback)
}
end (chunk, encoding, callback) {
console.log('END', chunk, encoding)
return super.end(chunk, encoding, callback)
}
}
someSource.pipe(new Logger()).pipe(someDest)
```
### same thing, but using an inline anonymous class
```js
// js classes are fun
someSource
.pipe(new (class extends Minipass {
emit (ev, ...data) {
// let's also log events, because debugging some weird thing
console.log('EMIT', ev)
return super.emit(ev, ...data)
}
write (chunk, encoding, callback) {
console.log('WRITE', chunk, encoding)
return super.write(chunk, encoding, callback)
}
end (chunk, encoding, callback) {
console.log('END', chunk, encoding)
return super.end(chunk, encoding, callback)
}
}))
.pipe(someDest)
```
### subclass that defers 'end' for some reason
```js
class SlowEnd extends Minipass {
emit (ev, ...args) {
if (ev === 'end') {
console.log('going to end, hold on a sec')
setTimeout(() => {
console.log('ok, ready to end now')
super.emit('end', ...args)
}, 100)
} else {
return super.emit(ev, ...args)
}
}
}
```
### transform that creates newline-delimited JSON
```js
class NDJSONEncode extends Minipass {
write (obj, cb) {
try {
// JSON.stringify can throw, emit an error on that
return super.write(JSON.stringify(obj) + '\n', 'utf8', cb)
} catch (er) {
this.emit('error', er)
}
}
end (obj, cb) {
if (typeof obj === 'function') {
cb = obj
obj = undefined
}
if (obj !== undefined) {
this.write(obj)
}
return super.end(cb)
}
}
```
### transform that parses newline-delimited JSON
```js
class NDJSONDecode extends Minipass {
constructor (options) {
// always be in object mode, as far as Minipass is concerned
super({ objectMode: true })
this._jsonBuffer = ''
}
write (chunk, encoding, cb) {
if (typeof chunk === 'string' &&
typeof encoding === 'string' &&
encoding !== 'utf8') {
chunk = Buffer.from(chunk, encoding).toString()
} else if (Buffer.isBuffer(chunk))
chunk = chunk.toString()
}
if (typeof encoding === 'function') {
cb = encoding
}
const jsonData = (this._jsonBuffer + chunk).split('\n')
this._jsonBuffer = jsonData.pop()
for (let i = 0; i < jsonData.length; i++) {
try {
// JSON.parse can throw, emit an error on that
super.write(JSON.parse(jsonData[i]))
} catch (er) {
this.emit('error', er)
continue
}
}
if (cb)
cb()
}
}
```
@@ -0,0 +1,147 @@
/// <reference types="node" />
// Note: marking anything protected or private in the exported
// class will limit Minipass's ability to be used as the base
// for mixin classes.
import { EventEmitter } from 'events'
import { Stream } from 'stream'
declare namespace Minipass {
type Encoding = BufferEncoding | 'buffer' | null
interface Writable extends EventEmitter {
end(): any
write(chunk: any, ...args: any[]): any
}
interface Readable extends EventEmitter {
pause(): any
resume(): any
pipe(): any
}
type DualIterable<T> = Iterable<T> & AsyncIterable<T>
type ContiguousData = Buffer | ArrayBufferLike | ArrayBufferView | string
type BufferOrString = Buffer | string
interface StringOptions {
encoding: BufferEncoding
objectMode?: boolean
async?: boolean
}
interface BufferOptions {
encoding?: null | 'buffer'
objectMode?: boolean
async?: boolean
}
interface ObjectModeOptions {
objectMode: true
async?: boolean
}
interface PipeOptions {
end?: boolean
proxyErrors?: boolean
}
type Options<T> = T extends string
? StringOptions
: T extends Buffer
? BufferOptions
: ObjectModeOptions
}
declare class Minipass<
RType extends any = Buffer,
WType extends any = RType extends Minipass.BufferOrString
? Minipass.ContiguousData
: RType
>
extends Stream
implements Minipass.DualIterable<RType>
{
static isStream(stream: any): stream is Minipass.Readable | Minipass.Writable
readonly bufferLength: number
readonly flowing: boolean
readonly writable: boolean
readonly readable: boolean
readonly paused: boolean
readonly emittedEnd: boolean
readonly destroyed: boolean
/**
* Technically writable, but mutating it can change the type,
* so is not safe to do in TypeScript.
*/
readonly objectMode: boolean
async: boolean
/**
* Note: encoding is not actually read-only, and setEncoding(enc)
* exists. However, this type definition will insist that TypeScript
* programs declare the type of a Minipass stream up front, and if
* that type is string, then an encoding MUST be set in the ctor. If
* the type is Buffer, then the encoding must be missing, or set to
* 'buffer' or null. If the type is anything else, then objectMode
* must be set in the constructor options. So there is effectively
* no allowed way that a TS program can set the encoding after
* construction, as doing so will destroy any hope of type safety.
* TypeScript does not provide many options for changing the type of
* an object at run-time, which is what changing the encoding does.
*/
readonly encoding: Minipass.Encoding
// setEncoding(encoding: Encoding): void
// Options required if not reading buffers
constructor(
...args: RType extends Buffer
? [] | [Minipass.Options<RType>]
: [Minipass.Options<RType>]
)
write(chunk: WType, cb?: () => void): boolean
write(chunk: WType, encoding?: Minipass.Encoding, cb?: () => void): boolean
read(size?: number): RType
end(cb?: () => void): this
end(chunk: any, cb?: () => void): this
end(chunk: any, encoding?: Minipass.Encoding, cb?: () => void): this
pause(): void
resume(): void
promise(): Promise<void>
collect(): Promise<RType[]>
concat(): RType extends Minipass.BufferOrString ? Promise<RType> : never
destroy(er?: any): void
pipe<W extends Minipass.Writable>(dest: W, opts?: Minipass.PipeOptions): W
unpipe<W extends Minipass.Writable>(dest: W): void
/**
* alias for on()
*/
addEventHandler(event: string, listener: (...args: any[]) => any): this
on(event: string, listener: (...args: any[]) => any): this
on(event: 'data', listener: (chunk: RType) => any): this
on(event: 'error', listener: (error: any) => any): this
on(
event:
| 'readable'
| 'drain'
| 'resume'
| 'end'
| 'prefinish'
| 'finish'
| 'close',
listener: () => any
): this
[Symbol.iterator](): Iterator<RType>
[Symbol.asyncIterator](): AsyncIterator<RType>
}
export = Minipass
@@ -0,0 +1,657 @@
'use strict'
const proc = typeof process === 'object' && process ? process : {
stdout: null,
stderr: null,
}
const EE = require('events')
const Stream = require('stream')
const SD = require('string_decoder').StringDecoder
const EOF = Symbol('EOF')
const MAYBE_EMIT_END = Symbol('maybeEmitEnd')
const EMITTED_END = Symbol('emittedEnd')
const EMITTING_END = Symbol('emittingEnd')
const EMITTED_ERROR = Symbol('emittedError')
const CLOSED = Symbol('closed')
const READ = Symbol('read')
const FLUSH = Symbol('flush')
const FLUSHCHUNK = Symbol('flushChunk')
const ENCODING = Symbol('encoding')
const DECODER = Symbol('decoder')
const FLOWING = Symbol('flowing')
const PAUSED = Symbol('paused')
const RESUME = Symbol('resume')
const BUFFER = Symbol('buffer')
const PIPES = Symbol('pipes')
const BUFFERLENGTH = Symbol('bufferLength')
const BUFFERPUSH = Symbol('bufferPush')
const BUFFERSHIFT = Symbol('bufferShift')
const OBJECTMODE = Symbol('objectMode')
const DESTROYED = Symbol('destroyed')
const EMITDATA = Symbol('emitData')
const EMITEND = Symbol('emitEnd')
const EMITEND2 = Symbol('emitEnd2')
const ASYNC = Symbol('async')
const defer = fn => Promise.resolve().then(fn)
// TODO remove when Node v8 support drops
const doIter = global._MP_NO_ITERATOR_SYMBOLS_ !== '1'
const ASYNCITERATOR = doIter && Symbol.asyncIterator
|| Symbol('asyncIterator not implemented')
const ITERATOR = doIter && Symbol.iterator
|| Symbol('iterator not implemented')
// events that mean 'the stream is over'
// these are treated specially, and re-emitted
// if they are listened for after emitting.
const isEndish = ev =>
ev === 'end' ||
ev === 'finish' ||
ev === 'prefinish'
const isArrayBuffer = b => b instanceof ArrayBuffer ||
typeof b === 'object' &&
b.constructor &&
b.constructor.name === 'ArrayBuffer' &&
b.byteLength >= 0
const isArrayBufferView = b => !Buffer.isBuffer(b) && ArrayBuffer.isView(b)
class Pipe {
constructor (src, dest, opts) {
this.src = src
this.dest = dest
this.opts = opts
this.ondrain = () => src[RESUME]()
dest.on('drain', this.ondrain)
}
unpipe () {
this.dest.removeListener('drain', this.ondrain)
}
// istanbul ignore next - only here for the prototype
proxyErrors () {}
end () {
this.unpipe()
if (this.opts.end)
this.dest.end()
}
}
class PipeProxyErrors extends Pipe {
unpipe () {
this.src.removeListener('error', this.proxyErrors)
super.unpipe()
}
constructor (src, dest, opts) {
super(src, dest, opts)
this.proxyErrors = er => dest.emit('error', er)
src.on('error', this.proxyErrors)
}
}
module.exports = class Minipass extends Stream {
constructor (options) {
super()
this[FLOWING] = false
// whether we're explicitly paused
this[PAUSED] = false
this[PIPES] = []
this[BUFFER] = []
this[OBJECTMODE] = options && options.objectMode || false
if (this[OBJECTMODE])
this[ENCODING] = null
else
this[ENCODING] = options && options.encoding || null
if (this[ENCODING] === 'buffer')
this[ENCODING] = null
this[ASYNC] = options && !!options.async || false
this[DECODER] = this[ENCODING] ? new SD(this[ENCODING]) : null
this[EOF] = false
this[EMITTED_END] = false
this[EMITTING_END] = false
this[CLOSED] = false
this[EMITTED_ERROR] = null
this.writable = true
this.readable = true
this[BUFFERLENGTH] = 0
this[DESTROYED] = false
if (options && options.debugExposeBuffer === true) {
Object.defineProperty(this, 'buffer', { get: () => this[BUFFER] })
}
if (options && options.debugExposePipes === true) {
Object.defineProperty(this, 'pipes', { get: () => this[PIPES] })
}
}
get bufferLength () { return this[BUFFERLENGTH] }
get encoding () { return this[ENCODING] }
set encoding (enc) {
if (this[OBJECTMODE])
throw new Error('cannot set encoding in objectMode')
if (this[ENCODING] && enc !== this[ENCODING] &&
(this[DECODER] && this[DECODER].lastNeed || this[BUFFERLENGTH]))
throw new Error('cannot change encoding')
if (this[ENCODING] !== enc) {
this[DECODER] = enc ? new SD(enc) : null
if (this[BUFFER].length)
this[BUFFER] = this[BUFFER].map(chunk => this[DECODER].write(chunk))
}
this[ENCODING] = enc
}
setEncoding (enc) {
this.encoding = enc
}
get objectMode () { return this[OBJECTMODE] }
set objectMode (om) { this[OBJECTMODE] = this[OBJECTMODE] || !!om }
get ['async'] () { return this[ASYNC] }
set ['async'] (a) { this[ASYNC] = this[ASYNC] || !!a }
write (chunk, encoding, cb) {
if (this[EOF])
throw new Error('write after end')
if (this[DESTROYED]) {
this.emit('error', Object.assign(
new Error('Cannot call write after a stream was destroyed'),
{ code: 'ERR_STREAM_DESTROYED' }
))
return true
}
if (typeof encoding === 'function')
cb = encoding, encoding = 'utf8'
if (!encoding)
encoding = 'utf8'
const fn = this[ASYNC] ? defer : f => f()
// convert array buffers and typed array views into buffers
// at some point in the future, we may want to do the opposite!
// leave strings and buffers as-is
// anything else switches us into object mode
if (!this[OBJECTMODE] && !Buffer.isBuffer(chunk)) {
if (isArrayBufferView(chunk))
chunk = Buffer.from(chunk.buffer, chunk.byteOffset, chunk.byteLength)
else if (isArrayBuffer(chunk))
chunk = Buffer.from(chunk)
else if (typeof chunk !== 'string')
// use the setter so we throw if we have encoding set
this.objectMode = true
}
// handle object mode up front, since it's simpler
// this yields better performance, fewer checks later.
if (this[OBJECTMODE]) {
/* istanbul ignore if - maybe impossible? */
if (this.flowing && this[BUFFERLENGTH] !== 0)
this[FLUSH](true)
if (this.flowing)
this.emit('data', chunk)
else
this[BUFFERPUSH](chunk)
if (this[BUFFERLENGTH] !== 0)
this.emit('readable')
if (cb)
fn(cb)
return this.flowing
}
// at this point the chunk is a buffer or string
// don't buffer it up or send it to the decoder
if (!chunk.length) {
if (this[BUFFERLENGTH] !== 0)
this.emit('readable')
if (cb)
fn(cb)
return this.flowing
}
// fast-path writing strings of same encoding to a stream with
// an empty buffer, skipping the buffer/decoder dance
if (typeof chunk === 'string' &&
// unless it is a string already ready for us to use
!(encoding === this[ENCODING] && !this[DECODER].lastNeed)) {
chunk = Buffer.from(chunk, encoding)
}
if (Buffer.isBuffer(chunk) && this[ENCODING])
chunk = this[DECODER].write(chunk)
// Note: flushing CAN potentially switch us into not-flowing mode
if (this.flowing && this[BUFFERLENGTH] !== 0)
this[FLUSH](true)
if (this.flowing)
this.emit('data', chunk)
else
this[BUFFERPUSH](chunk)
if (this[BUFFERLENGTH] !== 0)
this.emit('readable')
if (cb)
fn(cb)
return this.flowing
}
read (n) {
if (this[DESTROYED])
return null
if (this[BUFFERLENGTH] === 0 || n === 0 || n > this[BUFFERLENGTH]) {
this[MAYBE_EMIT_END]()
return null
}
if (this[OBJECTMODE])
n = null
if (this[BUFFER].length > 1 && !this[OBJECTMODE]) {
if (this.encoding)
this[BUFFER] = [this[BUFFER].join('')]
else
this[BUFFER] = [Buffer.concat(this[BUFFER], this[BUFFERLENGTH])]
}
const ret = this[READ](n || null, this[BUFFER][0])
this[MAYBE_EMIT_END]()
return ret
}
[READ] (n, chunk) {
if (n === chunk.length || n === null)
this[BUFFERSHIFT]()
else {
this[BUFFER][0] = chunk.slice(n)
chunk = chunk.slice(0, n)
this[BUFFERLENGTH] -= n
}
this.emit('data', chunk)
if (!this[BUFFER].length && !this[EOF])
this.emit('drain')
return chunk
}
end (chunk, encoding, cb) {
if (typeof chunk === 'function')
cb = chunk, chunk = null
if (typeof encoding === 'function')
cb = encoding, encoding = 'utf8'
if (chunk)
this.write(chunk, encoding)
if (cb)
this.once('end', cb)
this[EOF] = true
this.writable = false
// if we haven't written anything, then go ahead and emit,
// even if we're not reading.
// we'll re-emit if a new 'end' listener is added anyway.
// This makes MP more suitable to write-only use cases.
if (this.flowing || !this[PAUSED])
this[MAYBE_EMIT_END]()
return this
}
// don't let the internal resume be overwritten
[RESUME] () {
if (this[DESTROYED])
return
this[PAUSED] = false
this[FLOWING] = true
this.emit('resume')
if (this[BUFFER].length)
this[FLUSH]()
else if (this[EOF])
this[MAYBE_EMIT_END]()
else
this.emit('drain')
}
resume () {
return this[RESUME]()
}
pause () {
this[FLOWING] = false
this[PAUSED] = true
}
get destroyed () {
return this[DESTROYED]
}
get flowing () {
return this[FLOWING]
}
get paused () {
return this[PAUSED]
}
[BUFFERPUSH] (chunk) {
if (this[OBJECTMODE])
this[BUFFERLENGTH] += 1
else
this[BUFFERLENGTH] += chunk.length
this[BUFFER].push(chunk)
}
[BUFFERSHIFT] () {
if (this[BUFFER].length) {
if (this[OBJECTMODE])
this[BUFFERLENGTH] -= 1
else
this[BUFFERLENGTH] -= this[BUFFER][0].length
}
return this[BUFFER].shift()
}
[FLUSH] (noDrain) {
do {} while (this[FLUSHCHUNK](this[BUFFERSHIFT]()))
if (!noDrain && !this[BUFFER].length && !this[EOF])
this.emit('drain')
}
[FLUSHCHUNK] (chunk) {
return chunk ? (this.emit('data', chunk), this.flowing) : false
}
pipe (dest, opts) {
if (this[DESTROYED])
return
const ended = this[EMITTED_END]
opts = opts || {}
if (dest === proc.stdout || dest === proc.stderr)
opts.end = false
else
opts.end = opts.end !== false
opts.proxyErrors = !!opts.proxyErrors
// piping an ended stream ends immediately
if (ended) {
if (opts.end)
dest.end()
} else {
this[PIPES].push(!opts.proxyErrors ? new Pipe(this, dest, opts)
: new PipeProxyErrors(this, dest, opts))
if (this[ASYNC])
defer(() => this[RESUME]())
else
this[RESUME]()
}
return dest
}
unpipe (dest) {
const p = this[PIPES].find(p => p.dest === dest)
if (p) {
this[PIPES].splice(this[PIPES].indexOf(p), 1)
p.unpipe()
}
}
addListener (ev, fn) {
return this.on(ev, fn)
}
on (ev, fn) {
const ret = super.on(ev, fn)
if (ev === 'data' && !this[PIPES].length && !this.flowing)
this[RESUME]()
else if (ev === 'readable' && this[BUFFERLENGTH] !== 0)
super.emit('readable')
else if (isEndish(ev) && this[EMITTED_END]) {
super.emit(ev)
this.removeAllListeners(ev)
} else if (ev === 'error' && this[EMITTED_ERROR]) {
if (this[ASYNC])
defer(() => fn.call(this, this[EMITTED_ERROR]))
else
fn.call(this, this[EMITTED_ERROR])
}
return ret
}
get emittedEnd () {
return this[EMITTED_END]
}
[MAYBE_EMIT_END] () {
if (!this[EMITTING_END] &&
!this[EMITTED_END] &&
!this[DESTROYED] &&
this[BUFFER].length === 0 &&
this[EOF]) {
this[EMITTING_END] = true
this.emit('end')
this.emit('prefinish')
this.emit('finish')
if (this[CLOSED])
this.emit('close')
this[EMITTING_END] = false
}
}
emit (ev, data, ...extra) {
// error and close are only events allowed after calling destroy()
if (ev !== 'error' && ev !== 'close' && ev !== DESTROYED && this[DESTROYED])
return
else if (ev === 'data') {
return !data ? false
: this[ASYNC] ? defer(() => this[EMITDATA](data))
: this[EMITDATA](data)
} else if (ev === 'end') {
return this[EMITEND]()
} else if (ev === 'close') {
this[CLOSED] = true
// don't emit close before 'end' and 'finish'
if (!this[EMITTED_END] && !this[DESTROYED])
return
const ret = super.emit('close')
this.removeAllListeners('close')
return ret
} else if (ev === 'error') {
this[EMITTED_ERROR] = data
const ret = super.emit('error', data)
this[MAYBE_EMIT_END]()
return ret
} else if (ev === 'resume') {
const ret = super.emit('resume')
this[MAYBE_EMIT_END]()
return ret
} else if (ev === 'finish' || ev === 'prefinish') {
const ret = super.emit(ev)
this.removeAllListeners(ev)
return ret
}
// Some other unknown event
const ret = super.emit(ev, data, ...extra)
this[MAYBE_EMIT_END]()
return ret
}
[EMITDATA] (data) {
for (const p of this[PIPES]) {
if (p.dest.write(data) === false)
this.pause()
}
const ret = super.emit('data', data)
this[MAYBE_EMIT_END]()
return ret
}
[EMITEND] () {
if (this[EMITTED_END])
return
this[EMITTED_END] = true
this.readable = false
if (this[ASYNC])
defer(() => this[EMITEND2]())
else
this[EMITEND2]()
}
[EMITEND2] () {
if (this[DECODER]) {
const data = this[DECODER].end()
if (data) {
for (const p of this[PIPES]) {
p.dest.write(data)
}
super.emit('data', data)
}
}
for (const p of this[PIPES]) {
p.end()
}
const ret = super.emit('end')
this.removeAllListeners('end')
return ret
}
// const all = await stream.collect()
collect () {
const buf = []
if (!this[OBJECTMODE])
buf.dataLength = 0
// set the promise first, in case an error is raised
// by triggering the flow here.
const p = this.promise()
this.on('data', c => {
buf.push(c)
if (!this[OBJECTMODE])
buf.dataLength += c.length
})
return p.then(() => buf)
}
// const data = await stream.concat()
concat () {
return this[OBJECTMODE]
? Promise.reject(new Error('cannot concat in objectMode'))
: this.collect().then(buf =>
this[OBJECTMODE]
? Promise.reject(new Error('cannot concat in objectMode'))
: this[ENCODING] ? buf.join('') : Buffer.concat(buf, buf.dataLength))
}
// stream.promise().then(() => done, er => emitted error)
promise () {
return new Promise((resolve, reject) => {
this.on(DESTROYED, () => reject(new Error('stream destroyed')))
this.on('error', er => reject(er))
this.on('end', () => resolve())
})
}
// for await (let chunk of stream)
[ASYNCITERATOR] () {
const next = () => {
const res = this.read()
if (res !== null)
return Promise.resolve({ done: false, value: res })
if (this[EOF])
return Promise.resolve({ done: true })
let resolve = null
let reject = null
const onerr = er => {
this.removeListener('data', ondata)
this.removeListener('end', onend)
reject(er)
}
const ondata = value => {
this.removeListener('error', onerr)
this.removeListener('end', onend)
this.pause()
resolve({ value: value, done: !!this[EOF] })
}
const onend = () => {
this.removeListener('error', onerr)
this.removeListener('data', ondata)
resolve({ done: true })
}
const ondestroy = () => onerr(new Error('stream destroyed'))
return new Promise((res, rej) => {
reject = rej
resolve = res
this.once(DESTROYED, ondestroy)
this.once('error', onerr)
this.once('end', onend)
this.once('data', ondata)
})
}
return { next }
}
// for (let chunk of stream)
[ITERATOR] () {
const next = () => {
const value = this.read()
const done = value === null
return { value, done }
}
return { next }
}
destroy (er) {
if (this[DESTROYED]) {
if (er)
this.emit('error', er)
else
this.emit(DESTROYED)
return this
}
this[DESTROYED] = true
// throw away all buffered data, it's never coming out
this[BUFFER].length = 0
this[BUFFERLENGTH] = 0
if (typeof this.close === 'function' && !this[CLOSED])
this.close()
if (er)
this.emit('error', er)
else // if no error to emit, still reject pending promises
this.emit(DESTROYED)
return this
}
static isStream (s) {
return !!s && (s instanceof Minipass || s instanceof Stream ||
s instanceof EE && (
typeof s.pipe === 'function' || // readable
(typeof s.write === 'function' && typeof s.end === 'function') // writable
))
}
}
@@ -0,0 +1,56 @@
{
"name": "minipass",
"version": "4.0.0",
"description": "minimal implementation of a PassThrough stream",
"main": "index.js",
"types": "index.d.ts",
"dependencies": {
"yallist": "^4.0.0"
},
"devDependencies": {
"@types/node": "^17.0.41",
"end-of-stream": "^1.4.0",
"prettier": "^2.6.2",
"tap": "^16.2.0",
"through2": "^2.0.3",
"ts-node": "^10.8.1",
"typescript": "^4.7.3"
},
"scripts": {
"test": "tap",
"preversion": "npm test",
"postversion": "npm publish",
"postpublish": "git push origin --follow-tags"
},
"repository": {
"type": "git",
"url": "git+https://github.com/isaacs/minipass.git"
},
"keywords": [
"passthrough",
"stream"
],
"author": "Isaac Z. Schlueter <i@izs.me> (http://blog.izs.me/)",
"license": "ISC",
"files": [
"index.d.ts",
"index.js"
],
"tap": {
"check-coverage": true
},
"engines": {
"node": ">=8"
},
"prettier": {
"semi": false,
"printWidth": 80,
"tabWidth": 2,
"useTabs": false,
"singleQuote": true,
"jsxSingleQuote": false,
"bracketSameLine": true,
"arrowParens": "avoid",
"endOfLine": "lf"
}
}
@@ -0,0 +1,15 @@
The ISC License
Copyright (c) npm, Inc.
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
@@ -0,0 +1,96 @@
# npm-package-arg
[![Build Status](https://travis-ci.org/npm/npm-package-arg.svg?branch=master)](https://travis-ci.org/npm/npm-package-arg)
Parses package name and specifier passed to commands like `npm install` or
`npm cache add`, or as found in `package.json` dependency sections.
## EXAMPLES
```javascript
var assert = require("assert")
var npa = require("npm-package-arg")
// Pass in the descriptor, and it'll return an object
try {
var parsed = npa("@bar/foo@1.2")
} catch (ex) {
…
}
```
## USING
`var npa = require('npm-package-arg')`
### var result = npa(*arg*[, *where*])
* *arg* - a string that you might pass to `npm install`, like:
`foo@1.2`, `@bar/foo@1.2`, `foo@user/foo`, `http://x.com/foo.tgz`,
`git+https://github.com/user/foo`, `bitbucket:user/foo`, `foo.tar.gz`,
`../foo/bar/` or `bar`. If the *arg* you provide doesn't have a specifier
part, eg `foo` then the specifier will default to `latest`.
* *where* - Optionally the path to resolve file paths relative to. Defaults to `process.cwd()`
**Throws** if the package name is invalid, a dist-tag is invalid or a URL's protocol is not supported.
### var result = npa.resolve(*name*, *spec*[, *where*])
* *name* - The name of the module you want to install. For example: `foo` or `@bar/foo`.
* *spec* - The specifier indicating where and how you can get this module. Something like:
`1.2`, `^1.7.17`, `http://x.com/foo.tgz`, `git+https://github.com/user/foo`,
`bitbucket:user/foo`, `file:foo.tar.gz` or `file:../foo/bar/`. If not
included then the default is `latest`.
* *where* - Optionally the path to resolve file paths relative to. Defaults to `process.cwd()`
**Throws** if the package name is invalid, a dist-tag is invalid or a URL's protocol is not supported.
### var purl = npa.toPurl(*arg*, *reg*)
Returns the [purl (package URL)](https://github.com/package-url/purl-spec) form of the given pacakge name/spec.
* *arg* - A package/version string. For example: `foo@1.0.0` or `@bar/foo@2.0.0-alpha.1`.
* *reg* - Optionally the URL to the package registry. If not specified, assumes the default
`https://registry.npmjs.org`.
**Throws** if the package name is invalid, or the supplied arg can't be resolved to a purl.
## RESULT OBJECT
The objects that are returned by npm-package-arg contain the following
keys:
* `type` - One of the following strings:
* `git` - A git repo
* `tag` - A tagged version, like `"foo@latest"`
* `version` - A specific version number, like `"foo@1.2.3"`
* `range` - A version range, like `"foo@2.x"`
* `file` - A local `.tar.gz`, `.tar` or `.tgz` file.
* `directory` - A local directory.
* `remote` - An http url (presumably to a tgz)
* `alias` - A specifier with an alias, like `myalias@npm:foo@1.2.3`
* `registry` - If true this specifier refers to a resource hosted on a
registry. This is true for `tag`, `version` and `range` types.
* `name` - If known, the `name` field expected in the resulting pkg.
* `scope` - If a name is something like `@org/module` then the `scope`
field will be set to `@org`. If it doesn't have a scoped name, then
scope is `null`.
* `escapedName` - A version of `name` escaped to match the npm scoped packages
specification. Mostly used when making requests against a registry. When
`name` is `null`, `escapedName` will also be `null`.
* `rawSpec` - The specifier part that was parsed out in calls to `npa(arg)`,
or the value of `spec` in calls to `npa.resolve(name, spec).
* `saveSpec` - The normalized specifier, for saving to package.json files.
`null` for registry dependencies.
* `fetchSpec` - The version of the specifier to be used to fetch this
resource. `null` for shortcuts to hosted git dependencies as there isn't
just one URL to try with them.
* `gitRange` - If set, this is a semver specifier to match against git tags with
* `gitCommittish` - If set, this is the specific committish to use with a git dependency.
* `hosted` - If `from === 'hosted'` then this will be a `hosted-git-info`
object. This property is not included when serializing the object as
JSON.
* `raw` - The original un-modified string that was provided. If called as
`npa.resolve(name, spec)` then this will be `name + '@' + spec`.
* `subSpec` - If `type === 'alias'`, this is a Result Object for parsing the
target specifier for the alias.
@@ -0,0 +1,431 @@
'use strict'
module.exports = npa
module.exports.resolve = resolve
module.exports.toPurl = toPurl
module.exports.Result = Result
const url = require('url')
const HostedGit = require('hosted-git-info')
const semver = require('semver')
const path = global.FAKE_WINDOWS ? require('path').win32 : require('path')
const validatePackageName = require('validate-npm-package-name')
const { homedir } = require('os')
const log = require('proc-log')
const isWindows = process.platform === 'win32' || global.FAKE_WINDOWS
const hasSlashes = isWindows ? /\\|[/]/ : /[/]/
const isURL = /^(?:git[+])?[a-z]+:/i
const isGit = /^[^@]+@[^:.]+\.[^:]+:.+$/i
const isFilename = /[.](?:tgz|tar.gz|tar)$/i
function npa (arg, where) {
let name
let spec
if (typeof arg === 'object') {
if (arg instanceof Result && (!where || where === arg.where)) {
return arg
} else if (arg.name && arg.rawSpec) {
return npa.resolve(arg.name, arg.rawSpec, where || arg.where)
} else {
return npa(arg.raw, where || arg.where)
}
}
const nameEndsAt = arg[0] === '@' ? arg.slice(1).indexOf('@') + 1 : arg.indexOf('@')
const namePart = nameEndsAt > 0 ? arg.slice(0, nameEndsAt) : arg
if (isURL.test(arg)) {
spec = arg
} else if (isGit.test(arg)) {
spec = `git+ssh://${arg}`
} else if (namePart[0] !== '@' && (hasSlashes.test(namePart) || isFilename.test(namePart))) {
spec = arg
} else if (nameEndsAt > 0) {
name = namePart
spec = arg.slice(nameEndsAt + 1) || '*'
} else {
const valid = validatePackageName(arg)
if (valid.validForOldPackages) {
name = arg
spec = '*'
} else {
spec = arg
}
}
return resolve(name, spec, where, arg)
}
const isFilespec = isWindows ? /^(?:[.]|~[/]|[/\\]|[a-zA-Z]:)/ : /^(?:[.]|~[/]|[/]|[a-zA-Z]:)/
function resolve (name, spec, where, arg) {
const res = new Result({
raw: arg,
name: name,
rawSpec: spec,
fromArgument: arg != null,
})
if (name) {
res.setName(name)
}
if (spec && (isFilespec.test(spec) || /^file:/i.test(spec))) {
return fromFile(res, where)
} else if (spec && /^npm:/i.test(spec)) {
return fromAlias(res, where)
}
const hosted = HostedGit.fromUrl(spec, {
noGitPlus: true,
noCommittish: true,
})
if (hosted) {
return fromHostedGit(res, hosted)
} else if (spec && isURL.test(spec)) {
return fromURL(res)
} else if (spec && (hasSlashes.test(spec) || isFilename.test(spec))) {
return fromFile(res, where)
} else {
return fromRegistry(res)
}
}
const defaultRegistry = 'https://registry.npmjs.org'
function toPurl (arg, reg = defaultRegistry) {
const res = npa(arg)
if (res.type !== 'version') {
throw invalidPurlType(res.type, res.raw)
}
// URI-encode leading @ of scoped packages
let purl = 'pkg:npm/' + res.name.replace(/^@/, '%40') + '@' + res.rawSpec
if (reg !== defaultRegistry) {
purl += '?repository_url=' + reg
}
return purl
}
function invalidPackageName (name, valid, raw) {
// eslint-disable-next-line max-len
const err = new Error(`Invalid package name "${name}" of package "${raw}": ${valid.errors.join('; ')}.`)
err.code = 'EINVALIDPACKAGENAME'
return err
}
function invalidTagName (name, raw) {
// eslint-disable-next-line max-len
const err = new Error(`Invalid tag name "${name}" of package "${raw}": Tags may not have any characters that encodeURIComponent encodes.`)
err.code = 'EINVALIDTAGNAME'
return err
}
function invalidPurlType (type, raw) {
// eslint-disable-next-line max-len
const err = new Error(`Invalid type "${type}" of package "${raw}": Purl can only be generated for "version" types.`)
err.code = 'EINVALIDPURLTYPE'
return err
}
function Result (opts) {
this.type = opts.type
this.registry = opts.registry
this.where = opts.where
if (opts.raw == null) {
this.raw = opts.name ? opts.name + '@' + opts.rawSpec : opts.rawSpec
} else {
this.raw = opts.raw
}
this.name = undefined
this.escapedName = undefined
this.scope = undefined
this.rawSpec = opts.rawSpec || ''
this.saveSpec = opts.saveSpec
this.fetchSpec = opts.fetchSpec
if (opts.name) {
this.setName(opts.name)
}
this.gitRange = opts.gitRange
this.gitCommittish = opts.gitCommittish
this.gitSubdir = opts.gitSubdir
this.hosted = opts.hosted
}
Result.prototype.setName = function (name) {
const valid = validatePackageName(name)
if (!valid.validForOldPackages) {
throw invalidPackageName(name, valid, this.raw)
}
this.name = name
this.scope = name[0] === '@' ? name.slice(0, name.indexOf('/')) : undefined
// scoped packages in couch must have slash url-encoded, e.g. @foo%2Fbar
this.escapedName = name.replace('/', '%2f')
return this
}
Result.prototype.toString = function () {
const full = []
if (this.name != null && this.name !== '') {
full.push(this.name)
}
const spec = this.saveSpec || this.fetchSpec || this.rawSpec
if (spec != null && spec !== '') {
full.push(spec)
}
return full.length ? full.join('@') : this.raw
}
Result.prototype.toJSON = function () {
const result = Object.assign({}, this)
delete result.hosted
return result
}
function setGitCommittish (res, committish) {
if (!committish) {
res.gitCommittish = null
return res
}
// for each :: separated item:
for (const part of committish.split('::')) {
// if the item has no : the n it is a commit-ish
if (!part.includes(':')) {
if (res.gitRange) {
throw new Error('cannot override existing semver range with a committish')
}
if (res.gitCommittish) {
throw new Error('cannot override existing committish with a second committish')
}
res.gitCommittish = part
continue
}
// split on name:value
const [name, value] = part.split(':')
// if name is semver do semver lookup of ref or tag
if (name === 'semver') {
if (res.gitCommittish) {
throw new Error('cannot override existing committish with a semver range')
}
if (res.gitRange) {
throw new Error('cannot override existing semver range with a second semver range')
}
res.gitRange = decodeURIComponent(value)
continue
}
if (name === 'path') {
if (res.gitSubdir) {
throw new Error('cannot override existing path with a second path')
}
res.gitSubdir = `/${value}`
continue
}
log.warn('npm-package-arg', `ignoring unknown key "${name}"`)
}
return res
}
function fromFile (res, where) {
if (!where) {
where = process.cwd()
}
res.type = isFilename.test(res.rawSpec) ? 'file' : 'directory'
res.where = where
// always put the '/' on where when resolving urls, or else
// file:foo from /path/to/bar goes to /path/to/foo, when we want
// it to be /path/to/bar/foo
let specUrl
let resolvedUrl
const prefix = (!/^file:/.test(res.rawSpec) ? 'file:' : '')
const rawWithPrefix = prefix + res.rawSpec
let rawNoPrefix = rawWithPrefix.replace(/^file:/, '')
try {
resolvedUrl = new url.URL(rawWithPrefix, `file://${path.resolve(where)}/`)
specUrl = new url.URL(rawWithPrefix)
} catch (originalError) {
const er = new Error('Invalid file: URL, must comply with RFC 8909')
throw Object.assign(er, {
raw: res.rawSpec,
spec: res,
where,
originalError,
})
}
// environment switch for testing
if (process.env.NPM_PACKAGE_ARG_8909_STRICT !== '1') {
// XXX backwards compatibility lack of compliance with 8909
// Remove when we want a breaking change to come into RFC compliance.
if (resolvedUrl.host && resolvedUrl.host !== 'localhost') {
const rawSpec = res.rawSpec.replace(/^file:\/\//, 'file:///')
resolvedUrl = new url.URL(rawSpec, `file://${path.resolve(where)}/`)
specUrl = new url.URL(rawSpec)
rawNoPrefix = rawSpec.replace(/^file:/, '')
}
// turn file:/../foo into file:../foo
// for 1, 2 or 3 leading slashes since we attempted
// in the previous step to make it a file protocol url with a leading slash
if (/^\/{1,3}\.\.?(\/|$)/.test(rawNoPrefix)) {
const rawSpec = res.rawSpec.replace(/^file:\/{1,3}/, 'file:')
resolvedUrl = new url.URL(rawSpec, `file://${path.resolve(where)}/`)
specUrl = new url.URL(rawSpec)
rawNoPrefix = rawSpec.replace(/^file:/, '')
}
// XXX end 8909 violation backwards compatibility section
}
// file:foo - relative url to ./foo
// file:/foo - absolute path /foo
// file:///foo - absolute path to /foo, no authority host
// file://localhost/foo - absolute path to /foo, on localhost
// file://foo - absolute path to / on foo host (error!)
if (resolvedUrl.host && resolvedUrl.host !== 'localhost') {
const msg = `Invalid file: URL, must be absolute if // present`
throw Object.assign(new Error(msg), {
raw: res.rawSpec,
parsed: resolvedUrl,
})
}
// turn /C:/blah into just C:/blah on windows
let specPath = decodeURIComponent(specUrl.pathname)
let resolvedPath = decodeURIComponent(resolvedUrl.pathname)
if (isWindows) {
specPath = specPath.replace(/^\/+([a-z]:\/)/i, '$1')
resolvedPath = resolvedPath.replace(/^\/+([a-z]:\/)/i, '$1')
}
// replace ~ with homedir, but keep the ~ in the saveSpec
// otherwise, make it relative to where param
if (/^\/~(\/|$)/.test(specPath)) {
res.saveSpec = `file:${specPath.substr(1)}`
resolvedPath = path.resolve(homedir(), specPath.substr(3))
} else if (!path.isAbsolute(rawNoPrefix)) {
res.saveSpec = `file:${path.relative(where, resolvedPath)}`
} else {
res.saveSpec = `file:${path.resolve(resolvedPath)}`
}
res.fetchSpec = path.resolve(where, resolvedPath)
return res
}
function fromHostedGit (res, hosted) {
res.type = 'git'
res.hosted = hosted
res.saveSpec = hosted.toString({ noGitPlus: false, noCommittish: false })
res.fetchSpec = hosted.getDefaultRepresentation() === 'shortcut' ? null : hosted.toString()
return setGitCommittish(res, hosted.committish)
}
function unsupportedURLType (protocol, spec) {
const err = new Error(`Unsupported URL Type "${protocol}": ${spec}`)
err.code = 'EUNSUPPORTEDPROTOCOL'
return err
}
function matchGitScp (spec) {
// git ssh specifiers are overloaded to also use scp-style git
// specifiers, so we have to parse those out and treat them special.
// They are NOT true URIs, so we can't hand them to `url.parse`.
//
// This regex looks for things that look like:
// git+ssh://git@my.custom.git.com:username/project.git#deadbeef
//
// ...and various combinations. The username in the beginning is *required*.
const matched = spec.match(/^git\+ssh:\/\/([^:#]+:[^#]+(?:\.git)?)(?:#(.*))?$/i)
return matched && !matched[1].match(/:[0-9]+\/?.*$/i) && {
fetchSpec: matched[1],
gitCommittish: matched[2] == null ? null : matched[2],
}
}
function fromURL (res) {
// eslint-disable-next-line node/no-deprecated-api
const urlparse = url.parse(res.rawSpec)
res.saveSpec = res.rawSpec
// check the protocol, and then see if it's git or not
switch (urlparse.protocol) {
case 'git:':
case 'git+http:':
case 'git+https:':
case 'git+rsync:':
case 'git+ftp:':
case 'git+file:':
case 'git+ssh:': {
res.type = 'git'
const match = urlparse.protocol === 'git+ssh:' ? matchGitScp(res.rawSpec)
: null
if (match) {
setGitCommittish(res, match.gitCommittish)
res.fetchSpec = match.fetchSpec
} else {
setGitCommittish(res, urlparse.hash != null ? urlparse.hash.slice(1) : '')
urlparse.protocol = urlparse.protocol.replace(/^git[+]/, '')
if (urlparse.protocol === 'file:' && /^git\+file:\/\/[a-z]:/i.test(res.rawSpec)) {
// keep the drive letter : on windows file paths
urlparse.host += ':'
urlparse.hostname += ':'
}
delete urlparse.hash
res.fetchSpec = url.format(urlparse)
}
break
}
case 'http:':
case 'https:':
res.type = 'remote'
res.fetchSpec = res.saveSpec
break
default:
throw unsupportedURLType(urlparse.protocol, res.rawSpec)
}
return res
}
function fromAlias (res, where) {
const subSpec = npa(res.rawSpec.substr(4), where)
if (subSpec.type === 'alias') {
throw new Error('nested aliases not supported')
}
if (!subSpec.registry) {
throw new Error('aliases only work for registry deps')
}
res.subSpec = subSpec
res.registry = true
res.type = 'alias'
res.saveSpec = null
res.fetchSpec = null
return res
}
function fromRegistry (res) {
res.registry = true
const spec = res.rawSpec.trim()
// no save spec for registry components as we save based on the fetched
// version, not on the argument so this can't compute that.
res.saveSpec = null
res.fetchSpec = spec
const version = semver.valid(spec, true)
const range = semver.validRange(spec, true)
if (version) {
res.type = 'version'
} else if (range) {
res.type = 'range'
} else {
if (encodeURIComponent(spec) !== spec) {
throw invalidTagName(spec, res.raw)
}
res.type = 'tag'
}
return res
}
@@ -0,0 +1,59 @@
{
"name": "npm-package-arg",
"version": "10.1.0",
"description": "Parse the things that can be arguments to `npm install`",
"main": "./lib/npa.js",
"directories": {
"test": "test"
},
"files": [
"bin/",
"lib/"
],
"dependencies": {
"hosted-git-info": "^6.0.0",
"proc-log": "^3.0.0",
"semver": "^7.3.5",
"validate-npm-package-name": "^5.0.0"
},
"devDependencies": {
"@npmcli/eslint-config": "^4.0.0",
"@npmcli/template-oss": "4.10.0",
"tap": "^16.0.1"
},
"scripts": {
"test": "tap",
"snap": "tap",
"npmclilint": "npmcli-lint",
"lint": "eslint \"**/*.js\"",
"lintfix": "npm run lint -- --fix",
"posttest": "npm run lint",
"postsnap": "npm run lintfix --",
"postlint": "template-oss-check",
"template-oss-apply": "template-oss-apply --force"
},
"repository": {
"type": "git",
"url": "https://github.com/npm/npm-package-arg.git"
},
"author": "GitHub Inc.",
"license": "ISC",
"bugs": {
"url": "https://github.com/npm/npm-package-arg/issues"
},
"homepage": "https://github.com/npm/npm-package-arg",
"engines": {
"node": "^14.17.0 || ^16.13.0 || >=18.0.0"
},
"tap": {
"branches": 97,
"nyc-arg": [
"--exclude",
"tap-snapshots/**"
]
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "4.10.0"
}
}
@@ -0,0 +1,15 @@
The ISC License
Copyright (c) GitHub, Inc.
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
@@ -0,0 +1,90 @@
# proc-log
Emits 'log' events on the process object which a log output listener can
consume and print to the terminal.
This is used by various modules within the npm CLI stack in order to send
log events that can be consumed by a listener on the process object.
## API
* `log.error(...args)` calls `process.emit('log', 'error', ...args)`
The highest log level. For printing extremely serious errors that
indicate something went wrong.
* `log.warn(...args)` calls `process.emit('log', 'warn', ...args)`
A fairly high log level. Things that the user needs to be aware of, but
which won't necessarily cause improper functioning of the system.
* `log.notice(...args)` calls `process.emit('log', 'notice', ...args)`
Notices which are important, but not necessarily dangerous or a cause for
excess concern.
* `log.info(...args)` calls `process.emit('log', 'info', ...args)`
Informative messages that may benefit the user, but aren't particularly
important.
* `log.verbose(...args)` calls `process.emit('log', 'verbose', ...args)`
Noisy output that is more detail that most users will care about.
* `log.silly(...args)` calls `process.emit('log', 'silly', ...args)`
Extremely noisy excessive logging messages that are typically only useful
for debugging.
* `log.http(...args)` calls `process.emit('log', 'http', ...args)`
Information about HTTP requests made and/or completed.
* `log.pause()` calls `process.emit('log', 'pause')` Used to tell
the consumer to stop printing messages.
* `log.resume()` calls `process.emit('log', 'resume')`
Used to tell the consumer that it is ok to print messages again.
* `log.LEVELS` an array of strings of all log method names
## Examples
Every method calls `process.emit('log', level, ...otherArgs)` internally.
So in order to consume those events you need to do `process.on('log', fn)`.
### Colorize based on level
Here's an example of how to consume `proc-log` events and colorize them
based on level:
```js
const chalk = require('chalk')
process.on('log', (level, ...args) => {
if (level === 'error') {
console.log(chalk.red(level), ...args)
} else {
console.log(chalk.blue(level), ...args)
}
})
```
### Pause and resume
`pause` and `resume` are included so you have the ability to tell your consumer
that you want to pause or resume your display of logs. In the npm CLI we use
this to buffer all logs on init until we know the correct loglevel to display.
But we also setup a second handler that writes everything to a file even if
paused.
```js
let paused = true
const buffer = []
// this handler will buffer and replay logs only
// after `procLog.resume()` is called
process.on('log', (level, ...args) => {
if (level === 'resume') {
buffer.forEach((item) => console.log(...item))
paused = false
return
} 
if (paused) {
buffer.push([level, ...args])
} else {
console.log(level, ...args)
}
})
// this handler will write everything to a file
process.on('log', (...args) => {
fs.appendFileSync('debug.log', args.join(' '))
})
```
@@ -0,0 +1,23 @@
// emits 'log' events on the process
const LEVELS = [
'notice',
'error',
'warn',
'info',
'verbose',
'http',
'silly',
'pause',
'resume',
]
const log = level => (...args) => process.emit('log', level, ...args)
const logger = {}
for (const level of LEVELS) {
logger[level] = log(level)
}
logger.LEVELS = LEVELS
module.exports = logger
@@ -0,0 +1,44 @@
{
"name": "proc-log",
"version": "3.0.0",
"files": [
"bin/",
"lib/"
],
"main": "lib/index.js",
"description": "just emit 'log' events on the process object",
"repository": {
"type": "git",
"url": "https://github.com/npm/proc-log.git"
},
"author": "GitHub Inc.",
"license": "ISC",
"scripts": {
"test": "tap",
"snap": "tap",
"posttest": "npm run lint",
"postsnap": "eslint index.js test/*.js --fix",
"lint": "eslint \"**/*.js\"",
"postlint": "template-oss-check",
"lintfix": "npm run lint -- --fix",
"template-oss-apply": "template-oss-apply --force"
},
"devDependencies": {
"@npmcli/eslint-config": "^3.0.1",
"@npmcli/template-oss": "4.5.1",
"tap": "^16.0.1"
},
"engines": {
"node": "^14.17.0 || ^16.13.0 || >=18.0.0"
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "4.5.1"
},
"tap": {
"nyc-arg": [
"--exclude",
"tap-snapshots/**"
]
}
}
@@ -0,0 +1,6 @@
Copyright (c) 2015, npm, Inc
Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
@@ -0,0 +1,120 @@
# validate-npm-package-name
Give me a string and I'll tell you if it's a valid `npm` package name.
This package exports a single synchronous function that takes a `string` as
input and returns an object with two properties:
- `validForNewPackages` :: `Boolean`
- `validForOldPackages` :: `Boolean`
## Contents
- [Naming rules](#naming-rules)
- [Examples](#examples)
+ [Valid Names](#valid-names)
+ [Invalid Names](#invalid-names)
- [Legacy Names](#legacy-names)
- [Tests](#tests)
- [License](#license)
## Naming Rules
Below is a list of rules that valid `npm` package name should conform to.
- package name length should be greater than zero
- all the characters in the package name must be lowercase i.e., no uppercase or mixed case names are allowed
- package name *can* consist of hyphens
- package name must *not* contain any non-url-safe characters (since name ends up being part of a URL)
- package name should not start with `.` or `_`
- package name should *not* contain any spaces
- package name should *not* contain any of the following characters: `~)('!*`
- package name *cannot* be the same as a node.js/io.js core module nor a reserved/blacklisted name. For example, the following names are invalid:
+ http
+ stream
+ node_modules
+ favicon.ico
- package name length cannot exceed 214
## Examples
### Valid Names
```js
var validate = require("validate-npm-package-name")
validate("some-package")
validate("example.com")
validate("under_score")
validate("123numeric")
validate("@npm/thingy")
validate("@jane/foo.js")
```
All of the above names are valid, so you'll get this object back:
```js
{
validForNewPackages: true,
validForOldPackages: true
}
```
### Invalid Names
```js
validate("excited!")
validate(" leading-space:and:weirdchars")
```
That was never a valid package name, so you get this:
```js
{
validForNewPackages: false,
validForOldPackages: false,
errors: [
'name cannot contain leading or trailing spaces',
'name can only contain URL-friendly characters'
]
}
```
## Legacy Names
In the old days of npm, package names were wild. They could have capital
letters in them. They could be really long. They could be the name of an
existing module in node core.
If you give this function a package name that **used to be valid**, you'll see
a change in the value of `validForNewPackages` property, and a warnings array
will be present:
```js
validate("eLaBorAtE-paCkAgE-with-mixed-case-and-more-than-214-characters-----------------------------------------------------------------------------------------------------------------------------------------------------------")
```
returns:
```js
{
validForNewPackages: false,
validForOldPackages: true,
warnings: [
"name can no longer contain capital letters",
"name can no longer contain more than 214 characters"
]
}
```
## Tests
```sh
npm install
npm test
```
## License
ISC
@@ -0,0 +1,107 @@
'use strict'
var scopedPackagePattern = new RegExp('^(?:@([^/]+?)[/])?([^/]+?)$')
var builtins = require('builtins')
var blacklist = [
'node_modules',
'favicon.ico',
]
function validate (name) {
var warnings = []
var errors = []
if (name === null) {
errors.push('name cannot be null')
return done(warnings, errors)
}
if (name === undefined) {
errors.push('name cannot be undefined')
return done(warnings, errors)
}
if (typeof name !== 'string') {
errors.push('name must be a string')
return done(warnings, errors)
}
if (!name.length) {
errors.push('name length must be greater than zero')
}
if (name.match(/^\./)) {
errors.push('name cannot start with a period')
}
if (name.match(/^_/)) {
errors.push('name cannot start with an underscore')
}
if (name.trim() !== name) {
errors.push('name cannot contain leading or trailing spaces')
}
// No funny business
blacklist.forEach(function (blacklistedName) {
if (name.toLowerCase() === blacklistedName) {
errors.push(blacklistedName + ' is a blacklisted name')
}
})
// Generate warnings for stuff that used to be allowed
// core module names like http, events, util, etc
builtins({ version: '*' }).forEach(function (builtin) {
if (name.toLowerCase() === builtin) {
warnings.push(builtin + ' is a core module name')
}
})
if (name.length > 214) {
warnings.push('name can no longer contain more than 214 characters')
}
// mIxeD CaSe nAMEs
if (name.toLowerCase() !== name) {
warnings.push('name can no longer contain capital letters')
}
if (/[~'!()*]/.test(name.split('/').slice(-1)[0])) {
warnings.push('name can no longer contain special characters ("~\'!()*")')
}
if (encodeURIComponent(name) !== name) {
// Maybe it's a scoped package name, like @user/package
var nameMatch = name.match(scopedPackagePattern)
if (nameMatch) {
var user = nameMatch[1]
var pkg = nameMatch[2]
if (encodeURIComponent(user) === user && encodeURIComponent(pkg) === pkg) {
return done(warnings, errors)
}
}
errors.push('name can only contain URL-friendly characters')
}
return done(warnings, errors)
}
var done = function (warnings, errors) {
var result = {
validForNewPackages: errors.length === 0 && warnings.length === 0,
validForOldPackages: errors.length === 0,
warnings: warnings,
errors: errors,
}
if (!result.warnings.length) {
delete result.warnings
}
if (!result.errors.length) {
delete result.errors
}
return result
}
module.exports = validate
@@ -0,0 +1,65 @@
{
"name": "validate-npm-package-name",
"version": "5.0.0",
"description": "Give me a string and I'll tell you if it's a valid npm package name",
"main": "lib/",
"directories": {
"test": "test"
},
"dependencies": {
"builtins": "^5.0.0"
},
"devDependencies": {
"@npmcli/eslint-config": "^3.0.1",
"@npmcli/template-oss": "4.5.1",
"tap": "^16.0.1"
},
"scripts": {
"cov:test": "TAP_FLAGS='--cov' npm run test:code",
"test:code": "tap ${TAP_FLAGS:-'--'} test/*.js",
"test:style": "standard",
"test": "tap",
"lint": "eslint \"**/*.js\"",
"postlint": "template-oss-check",
"template-oss-apply": "template-oss-apply --force",
"lintfix": "npm run lint -- --fix",
"snap": "tap",
"posttest": "npm run lint"
},
"repository": {
"type": "git",
"url": "https://github.com/npm/validate-npm-package-name.git"
},
"keywords": [
"npm",
"package",
"names",
"validation"
],
"author": "GitHub Inc.",
"license": "ISC",
"bugs": {
"url": "https://github.com/npm/validate-npm-package-name/issues"
},
"homepage": "https://github.com/npm/validate-npm-package-name",
"files": [
"bin/",
"lib/"
],
"engines": {
"node": "^14.17.0 || ^16.13.0 || >=18.0.0"
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "4.5.1"
},
"tap": {
"statements": 88,
"branches": 92,
"lines": 88,
"nyc-arg": [
"--exclude",
"tap-snapshots/**"
]
}
}
+68
View File
@@ -0,0 +1,68 @@
{
"name": "npm-registry-fetch",
"version": "14.0.3",
"description": "Fetch-based http client for use with npm registry APIs",
"main": "lib",
"files": [
"bin/",
"lib/"
],
"scripts": {
"eslint": "eslint",
"lint": "eslint \"**/*.js\"",
"lintfix": "npm run lint -- --fix",
"test": "tap",
"posttest": "npm run lint",
"npmclilint": "npmcli-lint",
"postsnap": "npm run lintfix --",
"postlint": "template-oss-check",
"snap": "tap",
"template-oss-apply": "template-oss-apply --force"
},
"repository": {
"type": "git",
"url": "https://github.com/npm/npm-registry-fetch.git"
},
"keywords": [
"npm",
"registry",
"fetch"
],
"author": "GitHub Inc.",
"license": "ISC",
"dependencies": {
"make-fetch-happen": "^11.0.0",
"minipass": "^4.0.0",
"minipass-fetch": "^3.0.0",
"minipass-json-stream": "^1.0.1",
"minizlib": "^2.1.2",
"npm-package-arg": "^10.0.0",
"proc-log": "^3.0.0"
},
"devDependencies": {
"@npmcli/eslint-config": "^4.0.0",
"@npmcli/template-oss": "4.10.0",
"cacache": "^17.0.0",
"mkdirp": "^1.0.4",
"nock": "^13.2.4",
"require-inject": "^1.4.4",
"rimraf": "^3.0.2",
"ssri": "^10.0.0",
"tap": "^16.0.1"
},
"tap": {
"check-coverage": true,
"test-ignore": "test[\\\\/](util|cache)[\\\\/]",
"nyc-arg": [
"--exclude",
"tap-snapshots/**"
]
},
"engines": {
"node": "^14.17.0 || ^16.13.0 || >=18.0.0"
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "4.10.0"
}
}