Let's start from the very beginning: what exactly are web crawlers? Simply put, web crawlers, also known as spiders or bots, are automated programs that systematically browse the internet, collecting data from websites. They play a crucial role in indexing information for search engines like Google, enabling users to find relevant information quickly. However, their application extends far beyond search engines, finding use in data analysis, market research, and even in personal projects. Despite their utility, the operation of web crawlers comes with significant responsibilities and limitations, governed by both legal and ethical standards.

Legal Boundaries

The internet, often perceived as a vast and open space, is not exempt from legal regulations. Laws governing the use of web crawlers are akin to school rules for students. For instance, during exams, teachers enforce strict no-cheating policies. Similarly, laws prohibit unauthorized access to websites. In the United States, unauthorized data scraping can violate the Computer Fraud and Abuse Act (CFAA), leading to serious legal repercussions. In China, the legal landscape is somewhat more ambiguous, but the primary principles remain:

1. Do not access data without explicit permission.
2. Do not disrupt the normal operation of websites.

A landmark case highlighting these principles occurred in April 2019, when the Shenzhen Intermediate People's Court handled China's first "crawler" software lawsuit. The defendant, the developer of the app "Chelaile," was found guilty of using crawler technology to extract large amounts of data from a competitor, resulting in a court order to pay 500,000 RMB in damages.

Ethical Boundaries

While legal boundaries set the minimum standards for behavior, ethical considerations often serve as our moral compass. Imagine you're at a buffet: while it's permissible to take food, it's unethical to hoard more than you can eat, resulting in waste. Similarly, web crawlers can collect data, but they must do so responsibly. Overloading a website with requests can cause disruptions, like causing chaos at the buffet.

Privacy is another critical ethical consideration. Collecting personal information without consent is like peeping through someone's window with binoculars – it's invasive and illegal. Therefore, ensure that the data your crawler collects is public, and legal, and doesn't infringe on user privacy.

Responsible Use of Web Crawlers

To use web crawlers responsibly within the confines of law and ethics, consider the following best practices:

1. Respect Website Policies: Always read the robots.txt file of websites. This file acts as a guide, specifying which parts of the site can be accessed by crawlers and which parts are off-limits.
2. Limit Access Frequency: Avoid overloading websites by incorporating appropriate intervals between requests, mimicking the behavior of a human user.
3. Use Proxies and RPA Software: These tools can help distribute requests, reduce the risk of being blocked, and simulate normal user behavior.
4. Opt for Legal Data Acquisition: Whenever possible, acquire data through legitimate means. Purchasing data through proper channels is often safer and more reliable than risking legal consequences by scraping it.

Defending Against Crawlers

The relationship between crawlers and anti-crawling measures is a dynamic, cat-and-mouse game, much like the interplay between swords and shields. For engineers, defending against unauthorized crawlers is crucial. Here are some effective strategies:

1. Request Limiting and Denial: Implement rate limiting and access control measures to restrict the number of requests from a single source.
2. Client Authentication: Utilize methods like IP blocking and user-agent filtering to identify and block suspicious activity.
3. Text Obfuscation and Dynamic Rendering: Mix text with images and use CSS to obfuscate data, making it difficult for crawlers to parse.
4. Captchas: From basic text challenges to sophisticated puzzles, captchas are a well-known defense mechanism.

For example, Qunar.com employs CSS shifts to display ticket prices, complicating data extraction. Captchas have evolved from simple text-based challenges to complex spatial reasoning puzzles that even humans find difficult, illustrating the intensifying battle against crawlers.

How MoeGo Defending Against Crawlers

To address the issue of malicious data scraping and unauthorized submissions, we have implemented Google reCAPTCHA on each user's MoeGo Online Booking website. This powerful tool helps us distinguish between human users and automated bots, thereby enhancing the security and integrity of the booking process. By deploying reCAPTCHA, we ensure that the data entered and accessed on our platforms is protected against unwanted intrusions, providing our users with a safer and more reliable online experience.

Summary

Web crawlers are a double-edged sword. When used correctly, they can provide valuable insights and streamline information gathering. However, when misused, they can lead to legal troubles and ethical dilemmas. As engineers at MoeGo, it is imperative to master the technology while adhering to legal and ethical standards. By fostering a strong sense of security and responsibility in our development practices, we can become conscientious digital citizens, leveraging the power of web crawlers to enhance our work without crossing boundaries.

Author Bio

Bob (bob@moego.pet), from MoeGo R&D.

References

1. Web crawel definition
2. What is a web crawler? | How web spiders work
3. How to Crawl Responsibly: The Need for (Less) Speed
4. Robots.txt definition
5. Law and Ethics: The Boundaries of Web Crawlers

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License .

MoeGo mobile app offers in-person payment ability, powered by Stripe and Square, to our groommers. We integrate both the Stripe Terminal SDK and the Square Reader SDK, and they share some common stuffs which sometimes bring us problem.

When working on our latest release, we upgrades the Stripe Terminal React Native SDK to the latest version to support Tap to Pay on iPhone, where the native Stripe Terminal SDKs on both Android and iOS are upgraded. This works fine on iOS, but the app starts crashing on Android when connecting to Sqaure reader!

Inspect dependencies

Let's see the crash log first:

2023-04-26 13:15:07.059 23949-23949/? E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.moement.moego.business, PID: 23949
    java.lang.NoSuchMethodError: No static method createRuntimeMessageAdapter(Ljava/lang/Class;)Lcom/squareup/wire/internal/RuntimeMessageAdapter; in class Lcom/squareup/wire/internal/ReflectionKt; or its super classes (declaration of 'com.squareup.wire.internal.ReflectionKt' appears in /data/app/~~sA_8QsQq_POvPmiiiyky5Q==/com.moement.moego.business-egP-SXVX5YwQGid8AXqG7w==/base.apk!classes12.dex)
        at com.squareup.wire.WireTypeAdapterFactory.create(WireTypeAdapterFactory.kt:77)
        at com.google.gson.Gson.getAdapter(Gson.java:556)
        at retrofit2.converter.gson.GsonConverterFactory.requestBodyConverter(GsonConverterFactory.java:74)
        at retrofit2.Retrofit.nextRequestBodyConverter(Retrofit.java:315)
        at retrofit2.Retrofit.requestBodyConverter(Retrofit.java:293)
        at retrofit2.RequestFactory$Builder.parseParameterAnnotation(RequestFactory.java:778)
        at retrofit2.RequestFactory$Builder.parseParameter(RequestFactory.java:325)
        at retrofit2.RequestFactory$Builder.build(RequestFactory.java:206)
        at retrofit2.RequestFactory.parseAnnotations(RequestFactory.java:67)
        at retrofit2.ServiceMethod.parseAnnotations(ServiceMethod.java:26)
        at retrofit2.Retrofit.loadServiceMethod(Retrofit.java:202)
        at retrofit2.Retrofit$1.invoke(Retrofit.java:160)
        at java.lang.reflect.Proxy.invoke(Proxy.java:1006)
        at $Proxy7.login(Unknown Source)
        at com.squareup.sdk.reader.authorization.ReaderSdkAuthenticator.login(ReaderSdkAuthenticator.kt:116)
        at com.squareup.sdk.reader.authorization.RealAuthorizationManager.authorize(RealAuthorizationManager.java:109)
        at com.squareup.sdk.reader.react.AuthorizationModule$2.run(AuthorizationModule.java:117)
        at android.os.Handler.handleCallback(Handler.java:938)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:246)
        at android.app.ActivityThread.main(ActivityThread.java:8633)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:602)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1130)

To find out what's wrong, we should open the Android project with the Android Studio. Double-hit the Shift key and search for WireTypeAdapterFactory, which is complaining No static method createRuntimeMessageAdapter:

The WireTypeAdapterFactory.kt coming from wire-gson-support-3.7.0 is calling the createRuntimeMessageAdapter(Class<*>), which is provided in wire-runtime:3.7.0.

But wait, if we switch to the Project view, we will find that there are 2 wire-runtime dependencies of different versions. In wire-runtime-jvm:4.4.3, the single-parameter version of createRuntimeMessageAdapter is replaced by a two-parameter one!

In Java world, every class is given a package name, and two classes with the same name inside the same package lead to the compile error (the famous duplicate class error). That means we can't have two different versions of the same dependency in our app, though it is common in Node.js projects.

To figure out which version is chosen, we open the terminal and run the following command under the android directory:

./gradlew :app:dependencyInsight --configuration debugRuntimeClasspath --dependency wire-runtime

This will execute a Gradle task to tell us the detail about dependencies matching wire-runtime pattern. We choose debugRuntimeClasspath instead of debugCompileClasspath to get the full dependency tree vision.

com.squareup.wire:wire-runtime:4.4.3
+--- com.squareup.wire:wire-moshi-adapter:4.4.3
|    \--- com.stripe:stripeterminal-core:2.17.1
|         \--- com.stripe:stripeterminal:2.17.1
|              +--- debugRuntimeClasspath (requested com.stripe:stripeterminal:2.8.0)
|              \--- project :moego_stripe-terminal-react-native
|                   \--- debugRuntimeClasspath
\--- com.stripe:stripeterminal-core:2.17.1 (*)
​
com.squareup.wire:wire-runtime:2.2.0 -> 4.4.3
\--- com.squareup.retrofit2:converter-wire:2.7.2
     \--- com.squareup.sdk.reader:reader-sdk-xxxxxxxx:1.5.6
          \--- com.squareup.sdk.reader:reader-sdk-xxxxxxxx:1.5.6
               \--- project :react-native-square-reader-sdk
                    \--- debugRuntimeClasspath
​
com.squareup.wire:wire-runtime:3.7.0 -> 4.4.3
+--- com.squareup.sdk.reader:reader-sdk-xxxxxxxx:1.5.6
|    \--- com.squareup.sdk.reader:reader-sdk-xxxxxxxx:1.5.6
|         \--- project :react-native-square-reader-sdk
|              \--- debugRuntimeClasspath
\--- com.squareup.wire:wire-gson-support:3.7.0
     \--- com.squareup.sdk.reader:reader-sdk-xxxxxxxx:1.5.6 (*)

From the output we are told that wire-gson-support:3.7.0 depends on wire-runtime:3.7.0, but the latter is pulled up to 4.4.3 which breaks our app.

The normal approach: stick to the previous version

In most cases, we could just force Gradle to use the specific version of some dependency. We checkout the app code to the revision where the Tap to Pay on iPhone is not introduced yet, and run the dependency insight task again to find the "safe" version of wire-runtime: 4.0.0.

To lock a dependency to some version, edit android/app/build.gradle and add the following line to the dependencies block:

dependencies {
  implementation ('com.squareup.wire:wire-runtime:4.0.0') {
    version {
      strictly '4.0.0'
    }
  }
  // Other code...
}

Now the Square reader connection should work as expected, but if we are trying to discover the Stripe readers...

Caused by: java.lang.NoSuchMethodError: No static method createRuntimeMessageAdapter(Ljava/lang/Class;Z)Lcom/squareup/wire/internal/RuntimeMessageAdapter; in class Lcom/squareup/wire/internal/ReflectionKt; or its super classes (declaration of 'com.squareup.wire.internal.ReflectionKt' appears in /data/app/~~SjNakX7DciNr401J_dIdhA==/com.moement.moego.business-eVKkHNcBJoM42vrVJOJHng==/base.apk!classes30.dex)
                  at com.squareup.wire.WireJsonAdapterFactory.create(WireJsonAdapterFactory.kt:82)
                  at com.squareup.moshi.Moshi.adapter(Moshi.java:146)
                  at com.squareup.moshi.Moshi.adapter(Moshi.java:106)
                  at com.squareup.moshi.Moshi.adapter(Moshi.java:80)
                  at com.stripe.core.redaction.Extensions.toLogJson(Extensions.kt:20)
                  at com.stripe.core.redaction.Extensions.toLogJson$default(Extensions.kt:10)
                  at com.stripe.stripeterminal.internal.common.api.ApiLogPointInterceptor.intercept(ApiLogPointInterceptor.kt:26)
                  at com.stripe.core.restclient.InterceptorChain.proceed(InterceptorChain.kt:46)
                  at com.stripe.core.restclient.RestClient.execute(RestClient.kt:115)
                  at com.stripe.core.restclient.RestClient.blockingPost(RestClient.kt:57)
                  at com.stripe.proto.api.rest.MainlandApi.discoverLocations(MainlandApi.kt:336)
                  at com.stripe.proto.api.rest.MainlandApi.discoverLocations$default(MainlandApi.kt:327)
                  at com.stripe.core.transaction.AuthenticatedRestClient.discoverLocations(AuthenticatedRestClient.kt:233)
                  at com.stripe.stripeterminal.internal.common.api.ApiClient.discoverLocations(ApiClient.kt:162)
                  at com.stripe.stripeterminal.internal.common.resourcerepository.OnlineDirectResourceRepository.getReaderLocations(OnlineDirectResourceRepository.kt:832)
                  at com.stripe.stripeterminal.internal.common.resourcerepository.DirectResourceRepositoryRouter.getReaderLocations(DirectResourceRepositoryRouter.kt:173)
                  at com.stripe.stripeterminal.internal.common.resourcerepository.ProxyResourceRepository$getReaderLocations$1.invoke(ProxyResourceRepository.kt:126)
                  at com.stripe.stripeterminal.internal.common.resourcerepository.ProxyResourceRepository$getReaderLocations$1.invoke(ProxyResourceRepository.kt:126)
                  at com.stripe.stripeterminal.internal.common.resourcerepository.ProxyResourceRepository.withCurrentRepository(ProxyResourceRepository.kt:133)
                  at com.stripe.stripeterminal.internal.common.resourcerepository.ProxyResourceRepository.getReaderLocations(ProxyResourceRepository.kt:126)
                  at com.stripe.stripeterminal.internal.common.adapter.BbposAdapterLegacy.getLocationsForDiscovery(BbposAdapterLegacy.kt:805)
                  at com.stripe.stripeterminal.internal.common.adapter.BbposBluetoothAdapterLegacy.getLocationsForDiscovery(BbposBluetoothAdapterLegacy.kt:225)
15:43:24.244  E   at com.stripe.stripeterminal.internal.common.adapter.BbposBluetoothAdapterLegacy$DiscoverBluetoothReadersOperation.onUpdateDiscoveredReaders$lambda$6(BbposBluetoothAdapterLegacy.kt:577)
                  at com.stripe.stripeterminal.internal.common.adapter.BbposBluetoothAdapterLegacy$DiscoverBluetoothReadersOperation.lambda$OfuxcO_kCKP9oiD0iKlAxYfWlLs(Unknown Source:0)
                  at com.stripe.stripeterminal.internal.common.adapter.-$$Lambda$BbposBluetoothAdapterLegacy$DiscoverBluetoothReadersOperation$OfuxcO_kCKP9oiD0iKlAxYfWlLs.run(Unknown Source:6)
                  at io.reactivex.rxjava3.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:38)
                  ... 5 more

Ok, it turns out that Stripe Terminal SDK requires the latest version of wire-runtime to work properly!

Since the Tap to Pay on iPhone feature is only implemented by the iOS edition of the Stripe Terminal SDK, what about locking the version of Stripe Terminal Android SDK? In previous revision, our app depends on com.stripe:stripeterminal:2.14.0 which works well with wire-runtime:4.0.0.

We also give it a shot. This time the app doesn't crash, but behavior not very well... Open the Logcat view of Android Studio we may find some errors like this one:

16:33:30.975  I  Rejecting re-init on previously-failed class java.lang.Class<com.stripeterminalreactnative.MappersKt$WhenMappings>: java.lang.NoSuchFieldError: No static field WISEPAD_3S of type Lcom/stripe/stripeterminal/external/models/DeviceType; in class Lcom/stripe/stripeterminal/external/models/DeviceType; or its superclasses (declaration of 'com.stripe.stripeterminal.external.models.DeviceType' appears in /data/app/~~knBXOkTcLhdocfPMV5Um2g==/com.moement.moego.business-yk53x66eCZZZdbYD2X1n7w==/base.apk!classes31.dex)
                 (Throwable with no stack trace)
16:33:30.977  E  com.stripeterminalreactnative.MappersKt$WhenMappings
                 java.lang.NoClassDefFoundError: com.stripeterminalreactnative.MappersKt$WhenMappings
                  at com.stripeterminalreactnative.MappersKt.mapFromDeviceType(Mappers.kt:113)
                  at com.stripeterminalreactnative.MappersKt$mapFromReader$1.invoke(Mappers.kt:82)
                  at com.stripeterminalreactnative.MappersKt$mapFromReader$1.invoke(Mappers.kt:76)
                  at com.stripeterminalreactnative.MappersKt.nativeMapOf(Mappers.kt:63)
                  at com.stripeterminalreactnative.MappersKt.mapFromReader(Mappers.kt:76)
                  at com.stripeterminalreactnative.MappersKt$mapFromReaders$1.invoke(Mappers.kt:74)
                  at com.stripeterminalreactnative.MappersKt$mapFromReaders$1.invoke(Mappers.kt:74)
                  at com.stripeterminalreactnative.MappersKt.collectToWritableArray(Mappers.kt:448)
                  at com.stripeterminalreactnative.MappersKt.mapFromReaders(Mappers.kt:74)
                  at com.stripeterminalreactnative.listener.RNDiscoveryListener$onUpdateDiscoveredReaders$1.invoke(RNDiscoveryListener.kt:29)
                  at com.stripeterminalreactnative.listener.RNDiscoveryListener$onUpdateDiscoveredReaders$1.invoke(RNDiscoveryListener.kt:28)
                  at com.stripeterminalreactnative.ReactExtensions$sendEvent$1$1.invoke(ReactExtensions.kt:16)
                  at com.stripeterminalreactnative.ReactExtensions$sendEvent$1$1.invoke(ReactExtensions.kt:15)
                  at com.stripeterminalreactnative.MappersKt.nativeMapOf(Mappers.kt:63)
                  at com.stripeterminalreactnative.ReactExtensions.sendEvent(ReactExtensions.kt:15)
                  at com.stripeterminalreactnative.listener.RNDiscoveryListener.onUpdateDiscoveredReaders(RNDiscoveryListener.kt:28)
                  at com.stripe.stripeterminal.internal.common.callable.ProxyDiscoveryListener.onUpdateDiscoveredReaders(ProxyDiscoveryListener.kt:17)
                  at com.stripe.stripeterminal.adapter.RemoteReaderAdapter$DiscoverReadersOperation.execute(RemoteReaderAdapter.kt:153)
                  at com.stripe.stripeterminal.adapter.RemoteReaderAdapter$DiscoverReadersOperation.execute(RemoteReaderAdapter.kt:147)
                  at com.stripe.stripeterminal.adapter.RemoteReaderAdapter.discoverReaders(RemoteReaderAdapter.kt:128)
                  at com.stripe.stripeterminal.adapter.ProxyAdapter.discoverReaders(ProxyAdapter.kt:230)
                  at com.stripe.stripeterminal.TerminalSession$DiscoverReadersOperation.executeIfNotCanceled(TerminalSession.kt:1443)
                  at com.stripe.stripeterminal.TerminalSession$CancelableOperation.execute(TerminalSession.kt:693)
                  at com.stripe.stripeterminal.TerminalSession$ExternalOperation.run$core_publish(TerminalSession.kt:651)
                  at com.stripe.stripeterminal.TerminalSession.enqueueOperation$lambda-2(TerminalSession.kt:513)
                  at com.stripe.stripeterminal.TerminalSession.lambda$JON2uE3bqI6fqJ9Sw06Ciz8JRZw(Unknown Source:0)
                  at com.stripe.stripeterminal.-$$Lambda$TerminalSession$JON2uE3bqI6fqJ9Sw06Ciz8JRZw.run(Unknown Source:4)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:463)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:264)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637)
                  at java.lang.Thread.run(Thread.java:1012)
                 Caused by: java.lang.NoSuchFieldError: No static field WISEPAD_3S of type Lcom/stripe/stripeterminal/external/models/DeviceType; in class Lcom/stripe/stripeterminal/external/models/DeviceType; or its superclasses (declaration of 'com.stripe.stripeterminal.external.models.DeviceType' appears in /data/app/~~knBXOkTcLhdocfPMV5Um2g==/com.moement.moego.business-yk53x66eCZZZdbYD2X1n7w==/base.apk!classes31.dex)
                  at com.stripeterminalreactnative.MappersKt$WhenMappings.<clinit>(Unknown Source:135)
                  at com.stripeterminalreactnative.MappersKt.mapFromDeviceType(Mappers.kt:113) 
                  at com.stripeterminalreactnative.MappersKt$mapFromReader$1.invoke(Mappers.kt:82) 
                  at com.stripeterminalreactnative.MappersKt$mapFromReader$1.invoke(Mappers.kt:76) 
                  at com.stripeterminalreactnative.MappersKt.nativeMapOf(Mappers.kt:63) 
                  at com.stripeterminalreactnative.MappersKt.mapFromReader(Mappers.kt:76) 
                  at com.stripeterminalreactnative.MappersKt$mapFromReaders$1.invoke(Mappers.kt:74) 
                  at com.stripeterminalreactnative.MappersKt$mapFromReaders$1.invoke(Mappers.kt:74) 
                  at com.stripeterminalreactnative.MappersKt.collectToWritableArray(Mappers.kt:448) 
                  at com.stripeterminalreactnative.MappersKt.mapFromReaders(Mappers.kt:74) 
                  at com.stripeterminalreactnative.listener.RNDiscoveryListener$onUpdateDiscoveredReaders$1.invoke(RNDiscoveryListener.kt:29) 
                  at com.stripeterminalreactnative.listener.RNDiscoveryListener$onUpdateDiscoveredReaders$1.invoke(RNDiscoveryListener.kt:28) 
                  at com.stripeterminalreactnative.ReactExtensions$sendEvent$1$1.invoke(ReactExtensions.kt:16) 
                  at com.stripeterminalreactnative.ReactExtensions$sendEvent$1$1.invoke(ReactExtensions.kt:15) 
                  at com.stripeterminalreactnative.MappersKt.nativeMapOf(Mappers.kt:63) 
                  at com.stripeterminalreactnative.ReactExtensions.sendEvent(ReactExtensions.kt:15) 
                  at com.stripeterminalreactnative.listener.RNDiscoveryListener.onUpdateDiscoveredReaders(RNDiscoveryListener.kt:28) 
                  at com.stripe.stripeterminal.internal.common.callable.ProxyDiscoveryListener.onUpdateDiscoveredReaders(ProxyDiscoveryListener.kt:17) 
                  at com.stripe.stripeterminal.adapter.RemoteReaderAdapter$DiscoverReadersOperation.execute(RemoteReaderAdapter.kt:153) 
                  at com.stripe.stripeterminal.adapter.RemoteReaderAdapter$DiscoverReadersOperation.execute(RemoteReaderAdapter.kt:147) 
                  at com.stripe.stripeterminal.adapter.RemoteReaderAdapter.discoverReaders(RemoteReaderAdapter.kt:128) 
                  at com.stripe.stripeterminal.adapter.ProxyAdapter.discoverReaders(ProxyAdapter.kt:230) 
                  at com.stripe.stripeterminal.TerminalSession$DiscoverReadersOperation.executeIfNotCanceled(TerminalSession.kt:1443) 
                  at com.stripe.stripeterminal.TerminalSession$CancelableOperation.execute(TerminalSession.kt:693) 
                  at com.stripe.stripeterminal.TerminalSession$ExternalOperation.run$core_publish(TerminalSession.kt:651) 
                  at com.stripe.stripeterminal.TerminalSession.enqueueOperation$lambda-2(TerminalSession.kt:513) 
                  at com.stripe.stripeterminal.TerminalSession.lambda$JON2uE3bqI6fqJ9Sw06Ciz8JRZw(Unknown Source:0) 
                  at com.stripe.stripeterminal.-$$Lambda$TerminalSession$JON2uE3bqI6fqJ9Sw06Ciz8JRZw.run(Unknown Source:4) 
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:463) 
                  at java.util.concurrent.FutureTask.run(FutureTask.java:264) 
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137) 
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637) 
                  at java.lang.Thread.run(Thread.java:1012) 

That's because the Android native part of the Stripe Terminal React Native SDK depends on the updated Stripe Terminal Android SDK (only some enums, though). To fix this issue, we have to fork the RN SDK and drop those usages.

Locking com.stripe:stripeterminal's version has another great downside: we will never receive the latest features and bugfixes from the Stripe official!

The hacky approach: modify wire-gson-support

As we've learnt above, wire-gson-support:3.7.0, required by Square Reader SDK, depends on wire-runtime:3.7.0. And fortunately, the wire project is open source.

We clone the wire project and checkout the 3.7.0 tag.

On v4.4.3, here's the signature of the method:

// wire-library/wire-runtime/src/jvmMain/kotlin/com/squareup/wire/internal/reflection.kt
fun <M : Message<M, B>, B : Message.Builder<M, B>> createRuntimeMessageAdapter(
  messageType: Class<M>,
  writeIdentityValues: Boolean,
): RuntimeMessageAdapter<M, B> {
  // ...
}

There's only one added parameter. We copy the signature to the v3.7.0 code and update the usages. Then, run the Gradle task to build the jar file:

../gradlew :wire-gson-support:assemble

We will get the jar file under wire-library/wire-gson-support/build/libs/wire-gson-support-3.7.0.jar. The javadoc and sources jars can be safely ignored.

As the last step, we copy to jar file into our RN project's android/app/libs/ directory and tell Gradle not to use the original wire-gson-support (or we will get duplicate classes error!). Modify android/app/build.gradle and add:

configurations {
  implementation {
    exclude group: 'com.squareup.wire', module: 'wire-gson-support'
  }
}

Now build and test the app!

The caveat

We have to remember in mind that the solution above is just a workaround. Our app may still be broken if the new version of wire-runtime introduces some breaking change and the old wire-gson-support can't play with it. We need to carefully review all the usages of wire-runtime by wire-gson-support to ensure that all incompatibilities are handled. And this approach is only possible when we can easily grab the source code of some dependency.

Note: The following content uses MySQL version 8.0.24 and tables from the Sakila sample database.

Introduction

The Slow Query Log is a log used to record query statements that take a long time to execute in a MySQL database. In addition to recording DQL (Data Query Language), may also record DDL (Data Definition Language).

Specifically, when the query time is longer than long_query_time and the number of rows checked exceeds min_examined_row_limit, the query statement will be recorded in the slow query log.

By analyzing the slow query log, you can identify query statements that take longer to execute and optimize them to improve the performance of MySQL database.

Configuration Introduction

slow_query_log

Use slow_query_log to enable or disable slow query logging. The default is off.

Example usage:

SET GLOBAL slow_query_log = 'ON';

slow_query_log_file

Use the slow_query_log_file parameter to specify the output directory of the log. The default value is host_name-slow.log.

Example usage:

SET GLOBAL slow_query_log_file = '/www/server/data/mysql-slow.log';

long_query_time

the slow query threshold (in seconds) beyond which SQL execution will be logged. The default value is 10, the minimum value is 0, and the maximum value is 31536000 (365 days).

Example usage:

SET SESSION long_query_time = 1;

min_examined_row_limit

Slow query only logs SQL with scan rows greater than this parameter. only the number of rows checked at the server level is used here, not the number of rows scanned at the storage engine level. The default value is 0.

Example usage:

SET SESSION min_examined_row_limit = 100;

log_output

Use log_output to specify the log storage method. log_output='FILE' means to store the log to a file, the default value is 'FILE'. log_output='TABLE' means to store the log to the database so that the log information is written to the mysql.slow_log table. The MySQL database supports two types of log storage at the same time, just configure them separated by commas, for example: log_output='FILE,TABLE'. Logging to a dedicated system log table consumes more system resources than logging to a file, so for those who need to enable slow query logging and need to get higher system performance, it is recommended to prefer logging to a file.

Example usage:

SET GLOBAL log_output = 'FILE';

log_slow_admin_statements

log_slow_admin_statements can be used to log table operations, including ALTER TABLE, CREATE INDEX, and so on. The default is off.

Example usage:

SET GLOBAL log_slow_admin_statements = 'ON';

log_queries_not_using_indexes

log_queries_not_using_indexes records statements that do not use indexes. If the amount of data in the table is so small that a full table scan is faster and does not use the indexes, the same will be logged. The default value is off.

If only slow_query_log and log_queries_not_using_indexes are turned on, and the other parameters are the defaults, then the slow query log will log statements that use the UNION keyword, whether or not they hit an index. This appears to be a bug that is still not fixed in version 8.0.24: https://bugs.mysql.com/bug.php?id=29244.

Example usage:

SET GLOBAL log_queries_not_using_indexes = 'ON';

log_throttle_queries_not_using_indexes

log_throttle_queries_not_using_indexes to limit the number of statements that log no indexes for 60 seconds. The default is 0, which means no limit. Assuming a positive value of n, when the first statement without indexes is logged, MySQL opens a 60-second window where statements within the number n are logged and the rest are counted as a summary message.

Example usage:

SET GLOBAL log_throttle_queries_not_using_indexes = 200;

The above is modified by dynamic configuration and restarting either MySQL or the server will cause the configuration to fail. This can be done by writing the parameters to the configuration file to make it permanent.

Example configuration:

[mysqld]
slow_query_log=1
slow_query_log_file=/www/server/data/mysql-slow.log
long_query_time=1
min_examined_row_limit=100
log_queries_not_using_indexes=1

Record condition judgment

MySQL determines whether to log the current statement to the slow query log in the following order:

1. If it is an administrative statement and log_slow_admin_statements is turned on, the subsequent judgment is made, otherwise it is not logged.
2. If the query time is greater than long_query_time or log_queries_not_using_indexes is turned on and no indexes are used, then follow up, otherwise no logging is done.
3. If the query statement satisfies the number of rows set by at least min_examined_row_limit, it will be followed up, otherwise it will not be logged.
4. If log_throttle_queries_not_using_indexes is set and the current statement meets the required number of rows, it is logged, otherwise it is not logged.

With MySQL before 8.0, query statements that hit the cache are not logged to the slow query log.

Log File Format

Example of slow query log format:

# Time: 2023-05-29T09:09:28.842627Z
# User@Host: root[root] @ localhost []  Id:    12
# Query_time: 0.007624  Lock_time: 0.000363 Rows_sent: 1000  Rows_examined: 1006
SET timestamp=1685351368;
select *
from film f
         left join language l on f.language_id = l.language_id
where l.name = 'English';

The first line: the specific time when the SQL query was executed.

The second line: the connection information, user and connection IP of the SQL query execution.

The third line: Query_time indicates the time of SQL execution, the longer it is, the slower it is, Lock_time indicates the time of waiting for table lock in MySQL server phase (not in storage engine phase), Rows_sent indicates the number of rows returned by the query, Rows_examined indicates the number of rows checked by the query, the longer it is, the more time consuming it is.

The fourth line: set the timestamp, which has no practical meaning, but only corresponds to the execution time with the first line.

The fifth row and all subsequent rows are the executed SQL statements, and the SQL may be very long. Until the next # Time.

Slow Query Log Analysis Tools

Use MySQL's own slow query logging tool, mysqldumpslow, to filter and summarize data.

Commonly used parameters are explained:

-s sort_type

the type of sorting. The following options are available for sort_type: -s

• c: count Total number of executions
• l: lock time
• t: query time
• r: number of records returned
• al: average lock time
• at：Average query time
• ar：average number of returned records

-t N

returns the first N rows of data.

-g pattern

filter out the required information, you can write regular expressions, similar to grep command.

Example: Query the top 10 statements sorted by time that contain a left join:

mysqldumpslow -s t -t 10 -g "left join" /www/server/data/mysql-slow.log

Query results:

Reading mysql slow query log from /www/server/data/mysql-slow.log
Count: 1  Time=0.01s (0s)  Lock=0.00s (0s)  Rows=1000.0 (1000), root[root]@localhost
  select *
  from film f
  left join language l on f.language_id = l.language_id
  where l.name = 'S'

Died at /usr/bin/mysqldumpslow line 162, <> chunk 2.

Summary

The slow query log is a log used to record query statements that take a long time to execute in a MySQL database. By analyzing the information in the slow query log, you can find out the query statements that take longer to execute and perform targeted optimization to improve the performance of MySQL database.

Author Bio

Bob (bob@moego.pet), from MoeGo R&D.

References

1. https://dev.mysql.com/doc/sakila/en/
2. https://dev.mysql.com/doc/refman/8.0/en/slow-query-log.html
3. https://dev.mysql.com/doc/refman/8.0/en/mysqldumpslow.html
4. https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License .