perf(federation): Ed25519署名に対応する (#13464)

* 1. ed25519キーペアを発行・Personとして公開鍵を送受信 * validate additionalPublicKeys * getAuthUserFromApIdはmainを選ぶ * ✌️ * fix * signatureAlgorithm * set publicKeyCache lifetime * refresh * httpMessageSignatureAcceptable * ED25519_SIGNED_ALGORITHM * ED25519_PUBLIC_KEY_SIGNATURE_ALGORITHM * remove sign additionalPublicKeys signature requirements * httpMessageSignaturesSupported * httpMessageSignaturesImplementationLevel * httpMessageSignaturesImplementationLevel: '01' * perf(federation): Use hint for getAuthUserFromApId (#13470) * Hint for getAuthUserFromApId * とどのつまりこれでいいのか？ * use @misskey-dev/node-http-message-signatures * fix * signedPost, signedGet * ap-request.tsを復活させる * remove digest prerender * fix test? * fix test * add httpMessageSignaturesImplementationLevel to FederationInstance * ManyToOne * fetchPersonWithRenewal * exactKey * ✌️ * use const * use gen-key-pair fn. from '@misskey-dev/node-http-message-signatures' * update node-http-message-signatures * fix * @misskey-dev/node-http-message-signatures@0.0.0-alpha.11 * getAuthUserFromApIdでupdatePersonの頻度を増やす * cacheRaw.date * use requiredInputs https://github.com/misskey-dev/misskey/pull/13464#discussion_r1509964359 * update @misskey-dev/node-http-message-signatures * clean up * err msg * fix(backend): fetchInstanceMetadataのLockが永遠に解除されない問題を修正 Co-authored-by: まっちゃとーにゅ <17376330+u1-liquid@users.noreply.github.com> * fix httpMessageSignaturesImplementationLevel validation * fix test * fix * comment * comment * improve test * fix * use Promise.all in genRSAAndEd25519KeyPair * refreshAndprepareEd25519KeyPair * refreshAndfindKey * commetn * refactor public keys add * digestプリレンダを復活させる RFC実装時にどうするか考える * fix, async * fix * !== true * use save * Deliver update person when new key generated (not tested) https://github.com/misskey-dev/misskey/pull/13464#issuecomment-1977049061 * 循環参照で落ちるのを解消？ * fix? * Revert "fix?" This reverts commit 0082f6f8e8. * a * logger * log * change logger * 秘密鍵の変更は、フラグではなく鍵を引き回すようにする * addAllKnowingSharedInboxRecipe * nanka meccha kaeta * delivre * キャッシュ有効チェックはロック取得前に行う * @misskey-dev/node-http-message-signatures@0.0.3 * PrivateKeyPem * getLocalUserPrivateKey * fix test * if * fix ap-request * update node-http-message-signatures * fix type error * update package * fix type * update package * retry no key * @misskey-dev/node-http-message-signatures@0.0.8 * fix type error * log keyid * logger * db-resolver * JSON.stringify * HTTP Signatureがなかったり使えなかったりしそうな場合にLD Signatureを活用するように * inbox-delayed use actor if no signature * ユーザーとキーの同一性チェックはhostの一致にする * log signature parse err * save array * とりあえずtryで囲っておく * fetchPersonWithRenewalでエラーが起きたら古いデータを返す * use transactionalEntityManager * fix spdx * @misskey-dev/node-http-message-signatures@0.0.10 * add comment * fix * publicKeyに配列が入ってもいいようにする https://github.com/misskey-dev/misskey/pull/13950 * define additionalPublicKeys * fix * merge fix * refreshAndprepareEd25519KeyPair → refreshAndPrepareEd25519KeyPair * remove gen-key-pair.ts * defaultMaxListeners = 512 * Revert "defaultMaxListeners = 512" This reverts commit f2c412c180. * genRSAAndEd25519KeyPairではキーを直列に生成する? * maxConcurrency: 8 * maxConcurrency: 16 * maxConcurrency: 8 * Revert "genRSAAndEd25519KeyPairではキーを直列に生成する?" This reverts commit d0aada55c1. * maxWorkers: '90%' * Revert "maxWorkers: '90%'" This reverts commit 9e0a93f110. * e2e/timelines.tsで個々のテストに対するtimeoutを削除, maxConcurrency: 32 * better error handling of this.userPublickeysRepository.delete * better comment * set result to keypairEntityCache * deliverJobConcurrency: 16, deliverJobPerSec: 1024, inboxJobConcurrency: 4 * inboxJobPerSec: 64 * delete request.headers['host']; * fix * // node-fetch will generate this for us. if we keep 'Host', it won't change with redirects! * move delete host * modify comment * modify comment * fix correct → collect * refreshAndfindKey → refreshAndFindKey * modify comment * modify attachLdSignature * getApId, InboxProcessorService * TODO * [skip ci] add CHANGELOG --------- Co-authored-by: MeiMei <30769358+mei23@users.noreply.github.com> Co-authored-by: まっちゃとーにゅ <17376330+u1-liquid@users.noreply.github.com>
2024-07-18 01:28:17 +09:00
parent 3331f3972a
commit 5f88d56d96
52 changed files with 1096 additions and 694 deletions
--- a/packages/backend/src/core/FetchInstanceMetadataService.ts
+++ b/packages/backend/src/core/FetchInstanceMetadataService.ts
@@ -15,6 +15,7 @@ import { LoggerService } from '@/core/LoggerService.js';
 import { HttpRequestService } from '@/core/HttpRequestService.js';
 import { bindThis } from '@/decorators.js';
 import { FederatedInstanceService } from '@/core/FederatedInstanceService.js';
+import { REMOTE_SERVER_CACHE_TTL } from '@/const.js';
 import type { DOMWindow } from 'jsdom';

 type NodeInfo = {
@@ -24,6 +25,7 @@ type NodeInfo = {
 		version?: unknown;
 	};
 	metadata?: {
+		httpMessageSignaturesImplementationLevel?: unknown,
 		name?: unknown;
 		nodeName?: unknown;
 		nodeDescription?: unknown;
@@ -39,6 +41,7 @@ type NodeInfo = {
@Injectable()
 export class FetchInstanceMetadataService {
 	private logger: Logger;
+	private httpColon = 'https://';

 	constructor(
 		private httpRequestService: HttpRequestService,
@@ -48,6 +51,7 @@ export class FetchInstanceMetadataService {
 		private redisClient: Redis.Redis,
 	) {
 		this.logger = this.loggerService.getLogger('metadata', 'cyan');
+		this.httpColon = process.env.MISSKEY_USE_HTTP?.toLowerCase() === 'true' ? 'http://' : 'https://';
 	}

 	@bindThis
@@ -59,7 +63,7 @@ export class FetchInstanceMetadataService {
 		return await this.redisClient.set(
 			`fetchInstanceMetadata:mutex:v2:${host}`, '1',
 			'EX', 30, // 30秒したら自動でロック解除 https://github.com/misskey-dev/misskey/issues/13506#issuecomment-1975375395
-			'GET' // 古い値を返す（なかったらnull）
+			'GET', // 古い値を返す（なかったらnull）
 		);
 	}

@@ -73,23 +77,24 @@ export class FetchInstanceMetadataService {
 	public async fetchInstanceMetadata(instance: MiInstance, force = false): Promise<void> {
 		const host = instance.host;

-		// finallyでunlockされてしまうのでtry内でロックチェックをしない
-		// （returnであってもfinallyは実行される）
-		if (!force && await this.tryLock(host) === '1') {
-			// 1が返ってきていたらロックされているという意味なので、何もしない
-			return;
+		if (!force) {
+			// キャッシュ有効チェックはロック取得前に行う
+			const _instance = await this.federatedInstanceService.fetch(host);
+			const now = Date.now();
+			if (_instance && _instance.infoUpdatedAt != null && (now - _instance.infoUpdatedAt.getTime() < REMOTE_SERVER_CACHE_TTL)) {
+				this.logger.debug(`Skip because updated recently ${_instance.infoUpdatedAt.toJSON()}`);
+				return;
+			}
+
+			// finallyでunlockされてしまうのでtry内でロックチェックをしない
+			// （returnであってもfinallyは実行される）
+			if (await this.tryLock(host) === '1') {
+				// 1が返ってきていたら他にロックされているという意味なので、何もしない
+				return;
+			}
 		}

 		try {
-			if (!force) {
-				const _instance = await this.federatedInstanceService.fetch(host);
-				const now = Date.now();
-				if (_instance && _instance.infoUpdatedAt && (now - _instance.infoUpdatedAt.getTime() < 1000 * 60 * 60 * 24)) {
-					// unlock at the finally caluse
-					return;
-				}
-			}
-
 			this.logger.info(`Fetching metadata of ${instance.host} ...`);

 			const [info, dom, manifest] = await Promise.all([
@@ -118,6 +123,14 @@ export class FetchInstanceMetadataService {
 				updates.openRegistrations = info.openRegistrations;
 				updates.maintainerName = info.metadata ? info.metadata.maintainer ? (info.metadata.maintainer.name ?? null) : null : null;
 				updates.maintainerEmail = info.metadata ? info.metadata.maintainer ? (info.metadata.maintainer.email ?? null) : null : null;
+				if (info.metadata && info.metadata.httpMessageSignaturesImplementationLevel && (
+					info.metadata.httpMessageSignaturesImplementationLevel === '01' ||
+					info.metadata.httpMessageSignaturesImplementationLevel === '11'
+				)) {
+					updates.httpMessageSignaturesImplementationLevel = info.metadata.httpMessageSignaturesImplementationLevel;
+				} else {
+					updates.httpMessageSignaturesImplementationLevel = '00';
+				}
 			}

 			if (name) updates.name = name;
@@ -129,6 +142,12 @@ export class FetchInstanceMetadataService {
 			await this.federatedInstanceService.update(instance.id, updates);

 			this.logger.succ(`Successfuly updated metadata of ${instance.host}`);
+			this.logger.debug('Updated metadata:', {
+				info: !!info,
+				dom: !!dom,
+				manifest: !!manifest,
+				updates,
+			});
 		} catch (e) {
 			this.logger.error(`Failed to update metadata of ${instance.host}: ${e}`);
 		} finally {
@@ -141,7 +160,7 @@ export class FetchInstanceMetadataService {
 		this.logger.info(`Fetching nodeinfo of ${instance.host} ...`);

 		try {
-			const wellknown = await this.httpRequestService.getJson('https://' + instance.host + '/.well-known/nodeinfo')
+			const wellknown = await this.httpRequestService.getJson(this.httpColon + instance.host + '/.well-known/nodeinfo')
 				.catch(err => {
 					if (err.statusCode === 404) {
 						throw new Error('No nodeinfo provided');
@@ -184,7 +203,7 @@ export class FetchInstanceMetadataService {
 	private async fetchDom(instance: MiInstance): Promise<DOMWindow['document']> {
 		this.logger.info(`Fetching HTML of ${instance.host} ...`);

-		const url = 'https://' + instance.host;
+		const url = this.httpColon + instance.host;

 		const html = await this.httpRequestService.getHtml(url);

@@ -196,7 +215,7 @@ export class FetchInstanceMetadataService {

 	@bindThis
 	private async fetchManifest(instance: MiInstance): Promise<Record<string, unknown> | null> {
-		const url = 'https://' + instance.host;
+		const url = this.httpColon + instance.host;

 		const manifestUrl = url + '/manifest.json';

@@ -207,7 +226,7 @@ export class FetchInstanceMetadataService {

 	@bindThis
 	private async fetchFaviconUrl(instance: MiInstance, doc: DOMWindow['document'] | null): Promise<string | null> {
-		const url = 'https://' + instance.host;
+		const url = this.httpColon + instance.host;

 		if (doc) {
 			// https://github.com/misskey-dev/misskey/pull/8220#issuecomment-1025104043
@@ -234,12 +253,12 @@ export class FetchInstanceMetadataService {
 	@bindThis
 	private async fetchIconUrl(instance: MiInstance, doc: DOMWindow['document'] | null, manifest: Record<string, any> | null): Promise<string | null> {
 		if (manifest && manifest.icons && manifest.icons.length > 0 && manifest.icons[0].src) {
-			const url = 'https://' + instance.host;
+			const url = this.httpColon + instance.host;
 			return (new URL(manifest.icons[0].src, url)).href;
 		}

 		if (doc) {
-			const url = 'https://' + instance.host;
+			const url = this.httpColon + instance.host;

 			// https://github.com/misskey-dev/misskey/pull/8220#issuecomment-1025104043
 			const links = Array.from(doc.getElementsByTagName('link')).reverse();