Before writing a crawler, we need to capture packets of the target application and then analyze them before we can enter the stage of writing scripts.

For iPhone users, it's easy to capture packets on a daily basis. PC tools such as Charles and Fiddler are sufficient; Stream is a very powerful network packet capture application for iOS, with a simple interface and very powerful functions.

But for the use of Android high version of the system partners, grabbing packets is not so convenient! Because the security policy of the higher version system has been improved, the CA certificate must be placed in the system certificate directory in order to capture packets normally, and the certificate installed by the user is the user certificate by default, which makes it impossible for many apps to capture packets normally.

This article will introduce the operation steps of Android high version system packet capture with common scenarios

1.Non-Rooted Devices

Android also has a powerful network packet capture tool, namely: the small yellow bird "HttpCanary"

The specific operation steps are as follows.

1-1  Exporting HttpCanary Root Certificates

After installing the Little Yellow Bird App, go to the settings page and export the HttpCanary root certificate

click「 System Trusted(.0) 」

In this way, the HttpCanary root certificate is saved to the "internal storage path/HttpCanary/cert/.... .0" directory

1-2  Install APKPure and VMOS Pro applications

VMOS PRO download address:https://apkpure.com/cn/vmos-pro/com.vmos.ggp

PS: Since the VMOS PRO application format is XAPK, it is recommended to install the XAPK format application through APKPure.

1-3  VMOS import RE application, target application and HttpCanary root certificate

Open the VMOS Pro application and import the Root Explorer application, the target application and the HttpCanary root certificate file

This way, VMOS contains the target application, the RE file management application, and the HttpCanary root certificate file is saved under "VMOSfiletransferstatio/" by default

1-4  Import the certificate to the system certificate directory

In VMOS Pro, move the HttpCanary root certificate file to the system certificate directory via the RE application

System certificate directory: /system/etc/security/cacerts/

1-5  Start packet capture

Open the Little Yellow Bird App, set the target application as "VMSO" in the settings, then turn on the packet capture switch in the main interface, and finally operate the target application in VMOS.

The network requests of the target application will be displayed in the list of the main interface of Little Yellow Bird

2.Root device

If the phone is already rooted, we just need to move the third-party certificate (e.g., Little Yellow Bird, Charles, etc.) to the system certificate directory.

Here is the explanation by Little Yellow Bird App and Charles, Fiddler is similar

2-1  Little Yellow Bird App Grab Bag

The operation steps are as follows.

  • Unlock and Root your phone

  • Install the Little Yellow Bird HttpCanary application and export the HttpCanary root certificate, choosing the same format as above

  • Copy CA certificate to PC via data cable

  • Download adb and configure environment variables on PC

  • PUSH the certificate to the system certificate directory with the following series of adb commands

  • Open the Little Yellow Bird app and set the target app

  • Click the capture button in the main interface of Little Yellow Bird, that is, you can capture the packet of the target application

# Grant adb root privileges
adb root
# Disable system authentication
adb disable-verity 
# 
adb reboot

# Grant adb root privileges
adb root

# Before pushing the file to the '/system' folder, you must first enter the command 'adb remount'
adb remount

# Copy the certificate to /system/etc/security/cacerts/
# adb push 87bc3517.0 /system/etc/security/cacerts/

# Reboot
adb reroot

# Check if the imported CA certificate is included
adb root
adb shell
cd /system/etc/security/cacerts/
ls

2-2  Charles Capture Packet

The operation steps are as follows.

  • Charles download the certificate (e.g. CER certificate) in the help and copy it to the phone via data cable

  • Find this certificate in the file manager and install it manually

    It will be installed to the user certificate by default

    Installation directory:/data/misc/user/0/cacerts-added/

  • Use the following adb command to enable read and write access to the phone's system directory

  • Install the RE file management application and grant Root privileges, and move the above certificate from the user certificate directory to the system certificate directory

    System's certificate directory:/system/etc/security/cacerts

  • Reboot your phone

  • Packet Capture Test

    Check the ip address of the PC side, keep the phone on the same LAN, then set it as a manual proxy, and finally grab the packet for testing

# Execute with root privileges
adb root
# Disable system authentication
adb disable-verity

adb reboot
# Run with root privileges
adb root
# Remounting
adb remount

 

The above briefly illustrates the packet capture process for various scenarios of high version Android systems, based on whether the phone is Root or not.

In addition to the above packet capture methods, there are also many other options to choose from. For Root devices, we can install Magisk mask and use movecert module to capture packets, or we can use EdXposed framework + trustmealredy module to capture packets, in practice, we can choose the suitable way according to our needs

keywords: HttpCanary