Puppeteer - AWS Lambda 中使用 puppeteer 页面截图
in JavaScript with 0 comment

Puppeteer - AWS Lambda 中使用 puppeteer 页面截图

in JavaScript with 0 comment

之前公司项目,需要更换一个站点的后台系统,将整个站点迁移过去。

但是站点页面,包括各种新闻,超过8000多个。

人工进行新旧页面比对的话,emmmmm,大家也知道这种工程量。

于是决定做一个自动测试工具,使用 puppeteer 进行页面截图,然后对新旧站点页面截图比对,看新旧站点是否有样式缺失等情况。

并且为了节省服务器开支,我们需要将工具部署到 AWS 的 Lambda Functions 上,通过 API 进行访问,而截图都将存放在 AWS S3 中。

基于这个需求,我查阅了相关的资料,然后根据公司里其它前辈在 AWS Lambda 上的开发经验,实现了下面的功能。

(这里只提供 Puppeteer Screenshot in AWS Lambda 的实现代码,AWS API 会另外写文章)

Puppeteer: https://github.com/GoogleChrome/puppeteer

screenshot.js

'use strict';

const urlRequire = require('url');
const aws = require('aws-sdk');
const tmp = require('tmp');
const fs = require('fs');
const deviceDescriptors = require('puppeteer-core/DeviceDescriptors');
const AwsLambdaHttpResponse = require('aws-lambda-http-response');
const resolutionRepository = require('../repositories/repository_resolution').forOther;
const chromium = require('chrome-aws-lambda');
const puppeteer = require('puppeteer-core');
const utils = require('../utils/utils');

// Some parameters for puppeteer
const launchOptionForLambda = ['--no-sandbox', '--disable-gpu', '--single-process', '--disable-dev-shm-usage'];
const bugMaxHeight = 4000;

/**
 * @name handler
 * @description This function is open to Lambda
 * @param {*} event
 * @param {*} context
 * @param {*} callback
 * @return {AwsLambdaHttpResponse}
 */
exports.handler = async (event, context, callback) => {
  console.log('event', event);
  context.callbackWaitsForEmptyEventLoop = false;
  return exports
      .run(event)
      .then((result) => {
        return new AwsLambdaHttpResponse({callback}).success({
          body: result
        });
      })
      .catch((err) => {
        return new AwsLambdaHttpResponse({callback}).error({
          body: utils.errorFormatter(err).message
        });
      });
};

exports.run = (event) => {
  return new Promise((resolve, reject) => {
    if (!event.url) {
      console.log('URL was not provided.');
      return reject(new Error('URL was not provided.'));
    }

    const baseline = event.baseline == null ||
                     event.baseline == 'true' ||
                     event.baseline === true ? true : false;

    let devices = event.device == null ||
                  event.device == '' ? [] : [event.device];
    if (event.devices) devices = event.devices;

    const env = event.env == null ||
                event.env == '' ? null : event.env;

    return runScreenshot(event, baseline, devices, env)
        .then((result) => {
          return resolve({
            baseline: baseline,
            result: result
          });
        })
        .catch((err) => {
          console.log('handler', err);
          return reject(new Error(JSON.stringify({
            baseline: baseline,
            result: utils.errorFormatter(err).message
          })));
        });
  });
};

Function: runScreenshot

const runScreenshot = (event, baseline, devices, env) => {
  return new Promise((resolve, reject) => {
    const fnScreenshot = function fnScreenshot(device) {
      return new Promise((resolve, reject) => {
        const resObj = {};

        const user = event.user ? event.user : process.env.USER_AUTH;
        const password = event.password ? event.password : process.env.PASS_AUTH;

        const fnNavigate = async function fnNavigate(dv, url) {
          console.log('url', url);
          console.log('device', dv);

          let browser = null;

          try {
            browser = await puppeteer.launch({
              ignoreHTTPSErrors: true,
              args: launchOptionForLambda,
              executablePath: await chromium.executablePath,
              headless: true
            });

            const page = await browser.newPage();

            await page.setUserAgent(dv['userAgent']);

            await page.setViewport({
              width: dv['viewport']['width'],
              height: dv['viewport']['height'],
              deviceScaleFactor: dv['viewport']['deviceScaleFactor'],
              isMobile: dv['viewport']['isMobile'],
              hasTouch: dv['viewport']['hasTouch'],
              isLandscape: dv['viewport']['isLandscape']
            });

            const cookies = [
              {
                'url': url,
                'name': 'cookie_check',
                'value': 'approval'
              }
            ];

            await page.setCookie(...cookies);

            const auth = new Buffer(`${user}:${password}`).toString('base64');
            await page.setExtraHTTPHeaders({
              Authorization: `Basic ${auth}`
            });

            await page.goto(url, {waitUntil: 'networkidle0', timeout: 10000});

            if (!baseline) {
              await page.evaluate(() => {
                const style = document.createElement('style');
                style.textContent = `
                  #ReportWrapper { font-family: 'Noto Sans CJK JP Medium', Arial, Helvetica, sans-serif !important; }
                `;
                document.head.appendChild(style);
              });
            }

            if (baseline) {
              await page.addScriptTag({url: `${process.env.S3_INJECTION_URL}/custom.js`});
              await page.addScriptTag({url: `https://code.jquery.com/jquery-3.2.1.min.js`});
            }

            // Get page size
            const $ele = await page.$('html');
            const contentSize = await $ele.boundingBox();

            // Start screenshot
            const dpr = page.viewport().deviceScaleFactor || 1;
            const maxScreenshotHeight = Math.floor(bugMaxHeight / dpr);
            let img = '';
            const imgArr = [];
            if (contentSize.height < maxScreenshotHeight) {
              // Less than bug height
              console.log('small page');
              img = await page.screenshot({
                fullPage: true,
                encoding: 'binary'
              });
            } else {
              console.log('big page');
              for (let ypos = 0; ypos < contentSize.height; ypos += maxScreenshotHeight) {
                const height = Math.min(contentSize.height - ypos, maxScreenshotHeight);
                const tmpName = tmp.tmpNameSync();
                fs.writeFileSync(
                    tmpName,
                    await page.screenshot({
                      clip: {
                        x: 0,
                        y: ypos,
                        width: contentSize.width,
                        height: height
                      }
                    })
                );
                imgArr.push(tmpName);
              }
              img = await utils.imgMerger(imgArr);
            }

            await page.close();
            await browser.close();

            const thumbnail = await utils.imgThumbnail(img, 77, 144);

            const moveToS3 = [
              fnMoveToS3(thumbnail.toString('base64'), url, baseline, env, device, true),
              fnMoveToS3(img.toString('base64'), url, baseline, env, device, false)
            ];

            return Promise.all(moveToS3)
                .then((results) => {
                  resObj[device] = results;
                  return resolve(resObj);
                })
                .catch((err) => {
                  throw err;
                });
          } catch (e) {
            return reject(e);
          }
        };

        const url = event.url;

        if (deviceDescriptors[device] != null) {
          return fnNavigate(deviceDescriptors[device], url);
        } else {
          return resolutionRepository
              .getResolutionByName(device)
              .then((result) => {
                return fnNavigate(result, url);
              })
              .catch((err) => {
                return reject(err);
              });
        }
      });
    };

    const getScreenshots = devices.map(fnScreenshot);
    const results = Promise.all(getScreenshots);

    return results
        .then((res) => resolve(res))
        .catch((err) => reject(err));
  });
};

Function: fnMoveToS3

/**
 * @name fnMoveToS3
 * @description save img in S3
 * @param {String} img base64 image
 * @param {String} url path with host
 * @param {*} baseline true or false
 * @param {String} env site environment
 * @param {String} device device name
 * @param {Boolean} thumbnail image thumbnail
 * @return {Promise}
 */
const fnMoveToS3 = async function fnMoveToS3(img, url, baseline, env, device, thumbnail = false) {
  return new Promise((resolve, reject) => {
    const buf = new Buffer(img.replace(/^data:image\/\w+;base64,/, ''), 'base64');
    const urlString = url;
    const urlMap = urlRequire.parse(urlString, true).path;
    const folder = urlMap.replace(/\//g, '>');
    const s3 = new aws.S3({
      apiVersion: process.env.S3_API_VERSION
    });

    let fileName =
      (baseline ? 'baseline_' : 'current_' + env.trim() + '_') + device.trim() + '_screenshot.png';
    fileName = thumbnail ? 'thumbnail_' + fileName : fileName;
    console.log('fileName', fileName);

    return s3
        .putObject({
          Bucket: process.env.BUCKET_NAME,
          Key: folder + '/' + fileName,
          Body: buf,
          ContentType: 'image/png'
        })
        .promise()
        .then((data) => {
          return resolve('OK');
        })
        .catch((e) => {
          console.log('s3', e);
          return reject(e);
        });
  });
};

/**
 * @name validateParameters
 * @description Validate parameters required.
 * @param {Object} params Object with host_baseline, host_current, path or paths, devices and env required fields.
 * @return {Boolean}
 */
exports.validateParameters = function validateParameters(params) {
  if ( (!params.host_baseline && !params.host_current) ||
      (typeof params.baseline === 'undefined') ||
      (!params.path && !params.paths) ||
      (!params.device && !params.devices) ||
      !params.env) {
    return false;
  }
  return true;
};

custom_devices.js

module.exports = [
    {
        name: 'Desktop 1920x1080',
        userAgent:
            'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)' +
            ' Chrome/68.0.3440.75 Safari/537.36',
        viewport: {
            width: 1920,
            height: 1080,
            deviceScaleFactor: 1,
            isMobile: false,
            hasTouch: false,
            isLandscape: false
        }
    },
    {
        name: 'width: 768 x height: 1024',
        userAgent:
            'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)' +
            ' Chrome/68.0.3440.75 Safari/537.36',
        viewport: {
            width: 1024,
            height: 768,
            deviceScaleFactor: 1,
            isMobile: false,
            hasTouch: false,
            isLandscape: false
        }
    },
    {
        name: 'width: 1024 x height: 1366',
        userAgent:
            'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)' +
            ' Chrome/68.0.3440.75 Safari/537.36',
        viewport: {
            width: 1366,
            height: 1024,
            deviceScaleFactor: 1,
            isMobile: false,
            hasTouch: false,
            isLandscape: false
        }
    },
    {
        name: 'width: 375 x height: 812',
        userAgent:
            'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)' +
            ' Chrome/68.0.3440.75 Safari/537.36',
        viewport: {
            width: 375,
            height: 812,
            deviceScaleFactor: 1,
            isMobile: false,
            hasTouch: false,
            isLandscape: false
        }
    },
    {
        name: 'width: 750 x height: 1334',
        userAgent:
            'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)' +
            ' Chrome/68.0.3440.75 Safari/537.36',
        viewport: {
            width: 1334,
            height: 750,
            deviceScaleFactor: 1,
            isMobile: false,
            hasTouch: false,
            isLandscape: false
        }
    },
    {
        name: 'width: 1242 x height: 2688',
        userAgent:
            'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)' +
            ' Chrome/68.0.3440.75 Safari/537.36',
        viewport: {
            width: 1242,
            height: 2688,
            deviceScaleFactor: 1,
            isMobile: false,
            hasTouch: false,
            isLandscape: false
        }
    },
    {
        name: 'width: 1920 x height: 1080',
        userAgent:
            'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)' +
            ' Chrome/68.0.3440.75 Safari/537.36',
        viewport: {
            width: 1920,
            height: 1080,
            deviceScaleFactor: 1,
            isMobile: false,
            hasTouch: false,
            isLandscape: false
        }
    },
    {
        name: 'width: 1080 x height: 2160',
        userAgent:
            'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)' +
            ' Chrome/68.0.3440.75 Safari/537.36',
        viewport: {
            width: 1080,
            height: 2160,
            deviceScaleFactor: 1,
            isMobile: false,
            hasTouch: false,
            isLandscape: false
        }
    }
    ];
    for (const device of module.exports) {
        module.exports[device.name] = device;
    }

续篇

  1. 《Pixelmatch – AWS Lambda 中进行图片比较》
  2. 《Gm – 利用 gm 修改图片》
Responses