NodeJs

NodeJs - Puppeteer

Puppeteer

Puppeteer is a library that is used a lot by the Node.JS croud.

It allows you to manipulate web pages. There are lots of examples where puppeteer is used to scrape data out of websites.

Here is a sample code that works in Node.Js

1
2
3
4
5
6
7
8
9
10
11
12
13
const puppeteer = require('puppeteer'); 
  
(async () => { 
    const browser = await puppeteer.launch();
    console.log('1') ;
    const page = await browser.newPage(); 
    console.log('2') ;
    await page.goto('https://www.geeksforgeeks.org/'); 
    console.log('3') ;
    await browser.close(); 
    console.log('4') ;
    // this is working
})();

When I executed node myfirst.js it’s output included 1, 2, 3, 4 (each on separate lines)
At that point node kept running and did not stop until I pressed ^C

One oddity – it did not open a browser window like I expected. The behavior was the same both in a VSCode terminal window, and a Command window.

It turns out, puppeteer runs in two modes. One is called headless, the other is not. Headless means without UI. That is the default. If you want to see the browser window, then pass in { headless: false} as an argument to the puppeteer.launch method.

Note: Turning off headless works when I run from a console window, but I get a random error when I execute this function from the vsCode terminal window.

Node JS and Document.querySelectorAll(), I was not able to get this to work. I was able to get Document.querySector to work

Puppeteer object model

Page

Screenshot

Used to take a screen snapshot of a page, and save it
to the OS

Note: This would be as if you were opening a webpage and pressing the screen snapshot button, it’s not a full webpage, but rather a browser page.
Sample

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
const puppeteer = require('puppeteer'); 

let topic=’test’;
let folder=`./Mark-Lo-Pedia/${topic}`;
md(folder);

let waybackUrl=`http://web.archive.org/web/${theDate}/http://marklandcentral.com:80/wiki.aspx?topic=${topic}`;
console.log(`    WaybackUrl=${waybackUrl}`);

// goto website
const browser = await puppeteer.launch({ headless: false});
const page = await browser.newPage(); 
await page.goto(waybackUrl); // 4
   
// graphic snapshot
await page.screenshot({path: `${folder}/${topic}.png`})

// now close the browser
await browser.close();

References

Additional References