Before writing a crawler, we need to capture packets of the target application and then analyze them before we can enter the stage of writing scripts.
For iPhone users, it's easy to capture packets on a daily basis. PC tools such as Charles and Fiddler are sufficient; Stream is a very powerful network packet capture application for iOS, with a simple interface and very powerful functions.
But for the use of Android high version of the system partners, grabbing packets is not so convenient! Because the security policy of the higher version system has been improved, the CA certificate must be placed in the system certificate directory in order to capture packets normally, and the certificate installed by the user is the user certificate by default, which makes it impossible for many apps to capture packets normally.
This article will introduce the operation steps of Android high version system packet capture with common scenarios
Android also has a powerful network packet capture tool, namely: the small yellow bird "HttpCanary"
The specific operation steps are as follows.
1-1 Exporting HttpCanary Root Certificates
After installing the Little Yellow Bird App, go to the settings page and export the HttpCanary root certificate
click「 System Trusted(.0) 」
In this way, the HttpCanary root certificate is saved to the "internal storage path/HttpCanary/cert/.... .0" directory
1-2 Install APKPure and VMOS Pro applications
VMOS PRO download address：https://apkpure.com/cn/vmos-pro/com.vmos.ggp
PS: Since the VMOS PRO application format is XAPK, it is recommended to install the XAPK format application through APKPure.
1-3 VMOS import RE application, target application and HttpCanary root certificate
Open the VMOS Pro application and import the Root Explorer application, the target application and the HttpCanary root certificate file
This way, VMOS contains the target application, the RE file management application, and the HttpCanary root certificate file is saved under "VMOSfiletransferstatio/" by default
1-4 Import the certificate to the system certificate directory
In VMOS Pro, move the HttpCanary root certificate file to the system certificate directory via the RE application
System certificate directory: /system/etc/security/cacerts/
1-5 Start packet capture
Open the Little Yellow Bird App, set the target application as "VMSO" in the settings, then turn on the packet capture switch in the main interface, and finally operate the target application in VMOS.
The network requests of the target application will be displayed in the list of the main interface of Little Yellow Bird
If the phone is already rooted, we just need to move the third-party certificate (e.g., Little Yellow Bird, Charles, etc.) to the system certificate directory.
Here is the explanation by Little Yellow Bird App and Charles, Fiddler is similar
2-1 Little Yellow Bird App Grab Bag
The operation steps are as follows.
Unlock and Root your phone
Install the Little Yellow Bird HttpCanary application and export the HttpCanary root certificate, choosing the same format as above
Copy CA certificate to PC via data cable
Download adb and configure environment variables on PC
PUSH the certificate to the system certificate directory with the following series of adb commands
Open the Little Yellow Bird app and set the target app
Click the capture button in the main interface of Little Yellow Bird, that is, you can capture the packet of the target application
# Grant adb root privileges adb root # Disable system authentication adb disable-verity # adb reboot # Grant adb root privileges adb root # Before pushing the file to the '/system' folder, you must first enter the command 'adb remount' adb remount # Copy the certificate to /system/etc/security/cacerts/ # adb push 87bc3517.0 /system/etc/security/cacerts/ # Reboot adb reroot # Check if the imported CA certificate is included adb root adb shell cd /system/etc/security/cacerts/ ls
2-2 Charles Capture Packet
The operation steps are as follows.
Charles download the certificate (e.g. CER certificate) in the help and copy it to the phone via data cable
Find this certificate in the file manager and install it manually
It will be installed to the user certificate by default
Use the following adb command to enable read and write access to the phone's system directory
Install the RE file management application and grant Root privileges, and move the above certificate from the user certificate directory to the system certificate directory
System's certificate directory：/system/etc/security/cacerts
Reboot your phone
Packet Capture Test
Check the ip address of the PC side, keep the phone on the same LAN, then set it as a manual proxy, and finally grab the packet for testing
# Execute with root privileges adb root # Disable system authentication adb disable-verity # adb reboot # Run with root privileges adb root # Remounting adb remount
The above briefly illustrates the packet capture process for various scenarios of high version Android systems, based on whether the phone is Root or not.
In addition to the above packet capture methods, there are also many other options to choose from. For Root devices, we can install Magisk mask and use movecert module to capture packets, or we can use EdXposed framework + trustmealredy module to capture packets, in practice, we can choose the suitable way according to our needs