Thursday, September 11, 2025

Hello, my web automation framework!

Recently I’ve been enjoying web automation. If you often find yourself repeating the same manual steps in the browser — logging in, clicking menus, filling forms, checking results — you know how tedious it can get. To save time, I built a lightweight web automation framework using Node.js and Puppeteer. The best part is that I can pause the automation at any step, do some manual work in the browser if needed, and then continue with the following steps seamlessly.

Here are key design ideas:

1. Store inputs in .env
Sensitive values like username, password, and target URLs don’t belong in code. I load them at runtime with dotenv:
/* .env
USER=myuser
PASS=secretpassword
URL=https://example.com
*/

import dotenv from "dotenv";
dotenv.config();

console.log(process.env.USER, process.env.URL);
2. Share state with a global context object
Instead of passing around dozens of variables, I keep everything in one ctx object. This context is passed into all modules and steps, so they can use common objects like browser, page, or config flags.

3. Reuse or create a browser page
The framework can attach to an existing page of the Chrome session if it’s already there, or launch a new one if not. This way I can debug in a real browser window or let the script create a fresh page for me.

4. Define steps as reusable objects
All actions are defined in ctx.steps, where each step has a name and a run(ctx) function. For example:
// app.mjs
import { runner } from "../runner.mjs";

ctx.steps = [
  {
    name: "Page step 1",
    run: async (ctx) => {
      const { page } = ctx;
      // do something on page
    },
  },
  {
    name: "Page step 2",
    run: async (ctx) => {
      const { page } = ctx;
      // do something on page
    },
  },
];

// kick off the web automation
await runner(ctx);

5. Focus only on the steps.
With this framework, I just focus on writing the steps. And since they’re modular, I can run them sequentially, skip some, or even execute them in random order — all with the same framework.
Usage:
    node app.mjs                     # run all steps
    node app.mjs 3                   # run from step 3 to end
    node app.mjs 2 5                 # run steps 2..5
    node app.mjs --list              # list steps


In the future, I plan to extend this framework with screenshots on errors, step runtime tracking, and detailed reporting. That way, not only will the automation save me from repetitive work, but it will also give me clear insights into how each step performs and where failures happen.

Tuesday, September 2, 2025

Partial Web Automation: Chrome Debugging + Puppeteer

I don’t want to fully automate every interaction in the browser - just certain repetitive routines (e.g., filling in forms, checking values, etc.). At the same time, I like to watch the script run step by step so I can verify what’s happening.

Here’s how I’ve set things up to partially automate my workflow.
For example, when I need to log in to an insurance company’s website and fill out an application, I store the applicant information (name, birthday, address) in an Excel file. I log in to the site manually, then launch a Node.js script to handle the repetitive parts. This way, I can monitor each step as it runs and still perform the final review myself.

Steps:
1. Install Node.js and Puppeteer
# install the package globally  
npm install -g puppeteer 
2. Start Chrome with a debugging port
Launch Chrome with the option --remote-debugging-port=8888. To confirm it’s running correctly, check either of these URLs in your browser:
chrome://version/
http://127.0.0.1:8888/json/version (shows Chrome’s DevTools JSON endpoint)

3. Log in and let Puppeteer handle the routine
After logging in to the site manually, you can add code at the // DO SOMETHING placeholder to automate whichever steps you’d like.
const url = 'https://www.manulife.ca';
const puppeteer = require('puppeteer');

(async () => {
    // connect to Chrome
    const browser = await puppeteer.connect({
        browserURL: 'http://127.0.0.1:8888', // debug endpoint
        defaultViewport: null,
    });
    
    const pages = await browser.pages();
    console.log('NOTE: Pages currently open:', pages.map(p => p.url()));

    let page = pages.find(p => p.url().includes(url));

    if (page) {
        console.log('NOTE: Attaching existing page:', page.url());
        await page.bringToFront();
    } else {
        console.log('NOTE: Creating new one...');
        page = await browser.newPage();
        await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 90_000 });
    }

    // DO SOMETHING ...

    browser.disconnect();
})().catch(err => {
    console.log("ERROR occurs.");
    console.error(err);
    process.exit(1);
});

Saturday, May 17, 2025

Docker + Node.js + puppeteer: scrap Tugo travel insurance quote

I built a tool to scrap travel insurance quotes from various provider websites. For most of them, the process was straightforward — I could analyze the HTTP traffic to identify the relevant endpoints and use PROC HTTP to send POST requests directly to retrieve quotes.

However, scraping quotes from the TuGo website proved to be much more challenging.
TuGo’s quote system is highly dynamic and heavily reliant on JavaScript. Initially, I considered using Python with Selenium, but it quickly became clear that this approach wasn’t ideal due to performance limitations, maintenance problem and compatibility issues with the site’s rendering logic.

After reviewing several popular tools, I decided to go with Node.js and Puppeteer. This combination proved to be significantly more reliable and better suited for interacting with TuGo’s modern front-end framework.

Here are the basic steps I followed:
$ docker run -it \
  -v "$PWD:/home/pptruser/app" \
  -w /home/pptruser \
  --rm ghcr.io/puppeteer/puppeteer \
  node ./app/tugo_quote.js 06/07/1966 17/05/2025 31/05/2025
    
Sample output:
********  Inputs ********
traveller_dob: 06/07/1966
trip_start_date: 17/05/2025
trip_end_date: 31/05/2025
********  Tugo travel quote START ********
1. Basic information input
1.a Origin
1.b Destination
1.c Trip Start Date
1.d Trip End Date
1.e Trip Arrival Date
1.f Trip Cost
1.g Traveller info
1.h Click Button - Get a Quote
2. Quote results
2.a Close promotion dialog
2.b Fill in Questions if exist
No Questionnaire button found — skipping.
2.c Enable sliders
2.d Quote loop
Quote for Sum Insured $50K Deductiable $0 =  $60.48
Quote for Sum Insured $50K Deductiable $500 =  $54.43
Quote for Sum Insured $50K Deductiable $1000 =  $48.38
Quote for Sum Insured $100K Deductiable $0 =  $85.99
Quote for Sum Insured $100K Deductiable $500 =  $77.39
Quote for Sum Insured $100K Deductiable $1000 =  $68.80
******** Tugo travel quote END ********

Tuesday, April 29, 2025

How to download Google Sheet?

Google Drive is my faverite cloud storage. I save almost all my stuffs here. However, if the file is not public shared, you have to use OAuth 2 to access the file.

Below are the steps:
1. Create new Project in Google Cloud Console and setup OAuth 2.0 Client ID. Here we will set API scope for Google Drive and get client ID, client secret. Note that the information should be secured.

2. Get authorization code manually with URL below. Please open this in browser.
https://accounts.google.com/o/oauth2/auth?scope=https://www.googleapis.com/auth/drive&redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code&client_id=xxxxxxxx.apps.googleusercontent.com

3. Request for access token
%let auth_url=https://accounts.google.com/o/oauth2/auth;
%let redirect_uri=urn:ietf:wg:oauth:2.0:oob;
%let client_id=xxxxxxxxx.apps.googleusercontent.com; /* from step1 */
%let client_secret=xxxxxxxxxxx; /* from step1 */
%let code = xxxxxxxxxxx; /* from step2 */

proc http url="https://oauth2.googleapis.com/token"
    method="POST"
    out=resp
    headerout=hdrs
    ct="application/x-www-form-urlencoded"
    in=form(
            "code"="&code" 
            "client_id"="&client_id" 
            "client_secret"="&client_secret" 
            "redirect_uri"="&redirect_uri" 
            "grant_type"="authorization_code");
run;

%put INFO: &=SYS_PROCHTTP_STATUS_CODE;
%put INFO: &=SYS_PROCHTTP_STATUS_PHRASE;

4. Export the Google sheet as Excel file.
%let access_token=xxxxxxxxxxx; /* from step3 */
filename resp "download_path/result.xlsx";
proc http 
    url="https://docs.google.com/spreadsheets/d/2roVDi0WBqZ5t-gguUJ5eNKWSxJBP4AWiAc3e9sOgdtU/export?format=xlsx" 
    oauth_bear = "&access_token."
    out=resp;
run;

%put INFO: &=SYS_PROCHTTP_STATUS_CODE;
%put INFO: &=SYS_PROCHTTP_STATUS_PHRASE;

Sunday, January 19, 2025

Docker saspy

To keep my work environment clean, I always use docker to containize my faverite tool. I don't worry about the space as my work laptop has no any video. :) In my previous post, I have created script run_saspy.py to run SAS program locally. Below is my docker file to containize the script.
# Dockerfile
# Cmmand: docker build -t saspy .

FROM python

RUN apt update && apt install -y python3-pip && apt install -y default-jre
RUN pip install --upgrade pip
RUN pip install pandas
RUN pip install saspy

COPY sascfg_personal.py /usr/local/lib/python3.13/site-packages/saspy/sascfg_personal.py
COPY authinfo /root/.authinfo
COPY run_saspy.py /usr/local/bin/run_saspy.py

ENTRYPOINT ["run_saspy.py"]
When the docker image is built, I set alias run_saspy and it works perfectly.
$ alias run_saspy='docker run --rm -it -v ${PWD}:${PWD} -w "${PWD}" saspy -i '
$ run_saspy hello.sas
********************************************
Playpen: playpen
Using SAS Config named: oda
SAS Connection established. Subprocess id is 14

NOTE: hello.sas uploaded to ~/playpen/hello.sas successfully!
NOTE: ~/playpen/hello.20250120T003749.log downloaded to hello.20250120T003749.log successfully!
NOTE: Playpen killed successfully!
SAS Connection terminated. Subprocess id was 14
********************************************
********************************************
***        JOB EXECUTION STATUS          ***
********************************************
INFO: Finished successfully!
********************************************