Bridging the Gap between Reality and Ideality of Entity Matching: A Revisiting and Benchmark Re-Construction

1 Introduction

Entity resolution, also known as record linkage [Fellegi and Sunter, 1969] or deduplication [Meyer and Bolosky, 2012], is about merging records that refer to the same real-world entity. Entity matching (EM) aims to identify whether two entity records refer to the same real-world entity, which is the most critical step for entity resolution. For example, an effective entity matcher should resolve the two entity records "iPhone

∗Corresponding author.

13 Pro, CA" and "apple, iphone, 13 pro" into the same entity cluster because they refer to the same real-world product. Recent years have witnessed the impressive development of EM approaches, especially with the rapid progress of deep learning-based methods [Mudgal et al., 2018; Fu et al., 2020; Li et al., 2020]. Commonly, these methods are evaluated on the various EM benchmarks [Primpeli and Bizer, 2020] from different domains but constructed with a similar process. The state-of-the-art (SOTA) approaches have achieved over 90% F1 score on most of these datasets, which shows their strong effectiveness on these benchmarks.

However, once launched in real applications, the SOTA EM systems can become frustrating due to the discrepancies between current benchmarks and the open environment. Such discrepancies result in biased scores on the established benchmarks which can not well represent the real-world application performance. Unfortunately, there is no literature looking deep into this reality-ideality gap of entity matching, and therefore causes and effects of these discrepancies are not well studied. As a result, we are unable to evaluate the actual progress of novel approaches to real-world entity matching.

In this paper, we highlight that the discrepancies between benchmarks and the open environment stem from the implicit, unaware assumptions introduced during common benchmark construction. Such erroneous assumptions conceal the main challenges of entity matching, which lead to high performance on current benchmarks. Unfortunately, these assumptions do not hold in real-world scenarios, and therefore discrepancies between benchmarks and the open environment occur. Specifically, as illustrated in Figure 1, there are three critical erroneous assumptions made when constructing training and test sets of current EM benchmarks:

• Restricted Entity Assumption, which means that the entity clusters and/or entity records in the test sets of current benchmarks are mostly covered by records in the training sets, as shown in Table 1. This assumption is made because of the cluster/record sampling strategies applied for current benchmarks. Unfortunately, none of the mainstream entity resolution applications can meet, even be close to, such an assumption, because an effective entity matcher should be able to deal with unseen clusters and records in an open environment. As a result, current benchmarks are unable to evaluate entity matchers in an open environment.
• Balanced Label Assumption, which means that the ratios of mismatched to matched instances are relatively low and quite close in training and test sets of benchmarks. However, entity matching is an extremely imbalanced problem in real-world applications, and such the mismatched/matched ratio is commonly unknown and diverges significantly. Even with the prior blocking step, the ratio of mismatched record pairs to matched pairs can be up to 100:1 [Thirumuruganathan et al., 2021]. This poses the critical but ignored challenge to train and evaluate entity matchers in highly-imbalanced settings. As a result, current benchmarks are unable to evaluate entity matchers in the real-world imbalanced situation.
• Single Modality Assumption, which means current benchmarks mainly focus on textual attributes of entity records. This is due to the absence of the high-quality multi-modal entity corpus. However, in an open environment with noisy textual attributes, information from other modalities like images or audio can play a decisive role in entity matching. As a result, current benchmarks are difficult to be applied to accurately evaluate the effectiveness of multi-modal information for EM.

To assess how these three implicit assumptions bias evaluations of entity matchers, this paper constructs a new multi-modal corpus, which comes from a large Chinese ecommerce website and contains more than 120,000 records for 10,000 products. Each record in the corpus is with a highquality image attribute. Then starting from the corpus and standard benchmark construction criteria, we re-construct new benchmarks by removing the above-mentioned three erroneous assumptions step-by-step to revisit how these assumptions influence the evaluations on previous benchmarks. Specifically, for restricted entity assumption, we propose to leverage more practical cluster/record sampling strategies to build three benchmarks based on three most common EM application paradigms. For balanced label assumption, we vary the ratio of mismatched to matched instances on training and test set respectively to see how the real-world imbalanced situation can influence the evaluation. For single modality assumption, we thoroughly evaluate the effectiveness of introducing visual attributes based on the above real-world settings. From the newly re-constructed benchmarks, we find previous benchmarks are far from evaluating entity matching in an open environment because

• Restricted Entity Assumption biases the nature of the task of entity matching. The assumption changes the task from learning an effective matcher to learning effective representations of seen clusters/records. This discrepancy makes previous benchmarks significantly overestimate the performance of current entity matchers.
• Balanced Label Assumption conceals the most critical challenge of entity matching. We find that in realworld imbalanced scenarios, the performance on imbalanced test sets will dramatically diverge from the evaluation results on previous benchmarks, no matter how the degree of balance of the training set changes.
• Single Modality Assumption stems from the underestimation of the importance of multi-modality on previous benchmarks. We find that in the open environment, visual information can improve the performance of entity matching significantly. Moreover, the importance of visual information can significantly rise especially in open and imbalanced settings. This also confirms that previous restricted entities and balanced label benchmarks can not well estimate the effectiveness of multi-modal entity matchers.

Our re-constructed benchmark clearly shows that previous benchmark construction criteria can not cover the main challenges of entity matching, and there is still a long way to build an effective entity matcher in an open environment. Generally speaking, the main contributions of this paper are:

• We reveal three implicit assumptions behind current EM benchmarks. These assumptions are introduced during the benchmark construction process, which leads to significant discrepancies between current benchmarks and open environment.
• We build a new multi-modal entity matching corpus. The corpus contains more than 120,000 multimodal records for 10,000 products. This corpus provides

a solid foundation for evaluating the impact of above assumptions, as well as future research on multi-modal EM.

• We thoroughly assess the impact of the above three critical assumptions for evaluating entity matcher. Experimental results show that these assumptions conceal the most critical challenges of entity matching, and therefore significantly overestimate the current progress due to the biased benchmarks.

2 Background and NewCorpus Construction

In this section, we will first demonstrate the construction process of conventional entity matching benchmarks and point out three implicit assumptions made during the process. Then to assess the impacts of the three assumptions, we present a new multi-modal corpus for entity matching. Finally, we will briefly introduce the state-of-the-art approaches on entity matching, which will be used to evaluate the impact of the above-mentioned assumptions in the following sections.

2.1 Construction and Implicit Assumptions of Previous Benchmarks

Algorithm 1 outlines the construction process of current entity matching benchmarks. The procedure commonly first removes non-textual attributes from all records. Then a fixed number of entity clusters are sampled to construct datasets. Some record pairs within the same clusters are regarded as the matched entity record pairs, and a number of mismatched pairs are sampled from records from different clusters. The number of mismatched instances is commonly proportional to the number of matched instances with a fixed ratio. Finally, all record pairs with matched/mismatched labels are split into training/validation/test sets to build the standard benchmarks.

Such a construction process, however, is implicitly incorporated with three assumptions. First, because all records are sampled from the same group of clusters C ′ , there is a great chance that clusters and records in the test set would appear in the training set. To show this, Table 1 presents the statistics from several most widely-used EM benchmarks. We can see that nearly all entity clusters in the test set of these benchmarks are covered by the training set. Furthermore, a vast majority of entity records in the test set are also covered by the training set. However, in an open environment this assumption does not hold, because a great number of entity clusters and records are unseen during training. Second, due to the mismatched-matched instance sampling strategy, the ratio of mismatched pairs to matched pairs is relatively low, as shown in Table 1 again. Further, such ratios are nearly the same for the training and test set. Unfortunately, in real-world applications we will face an extremely high mismatched-matched ratio. Even after the blocking of entity resolution, a record may have up to 100 candidate matches. And due to the long-tail phenomenon, it is very frequently that only one instance among them is the matched one. Consequently, current benchmarks can not well reflect how well entity matchers can deal with such an extremely imbalanced situation. Third, many current benchmarks only focus on textual attributes, which multi-modal attributes become increas-

Data: A set of entity clusters C = {c1, c2, · · · , cn}, where each cluster ci contains several records ci = {r1 i , r2 i , · · · , rm i } Input: train/val/test ratio r, mismatched/matched ratio k Output: training, validation, and test set: train, val, test foreach ci ∈ C do foreach rj i ∈ ci do **Single modality assumption** preserving only textual attributes of record rj i end end sample a subset of clusters C ′ fromC D = Ø foreach ci ∈ C ′ do for rj i , rk i ∈ ci do D ← D ∪ (rj i , rk i ,matched) end end **Balanced label assumption** for i ← 1 to k ∗ ∥D∥ do sample different cluster records rli , rm j fromC ′ D ← D ∪ (rli , rm j ,mismatched) end **Restricted entity assumption** train, val, test ← RANDOM SPLIT(D, r) return train, val, test

ingly popular and important in many EM scenarios. Due to the above reasons, current benchmarks with these implicit assumptions can not well evaluate the performance of the entity matcher in the open environment.

2.2 NewCorpus Construction

In order to assess how these three implicit assumptions influence the evaluations of entity matchers, we construct a new multi-modal corpus based on product information from a Chinese e-commerce website. The newly built corpus contains 3 main categories (clothing, shoes, and accessories) of products. Each product record includes a record ID, title, categorical info, cluster ID, attribute pairs (e.g., color, style, material, etc.), as well as a product image. Entity records with the same cluster ID, which are manually checked, are considered as referring to the same real-world product entity. Finally, there are 10,000 products and 126,277 records in this corpus. Each product has records between 10 and 20.

From the corpus, we use the clusters from all categories and three specific categories to build the datasets. For each dataset, we randomly sample 250 clusters to build the training set and hold out 100 clusters for the unseen cluster benchmark. For each cluster in the training set, we also hold out 40% of the records as unseen records.

(a) Dataset Splits

(b) Benchmarks for different scenarios

2.3 Baselines

In this paper, we mainly focus on exploring the impact of the three above problems on deep learning-based approaches while leaving others for future work. We choose two representative methods as baselines in our experiments:

• Deepmatcher (DMatcher), which is the first detailed exploration of deep learning methods on EM [Mudgal et al., 2018]. We use their open source code directly.
• Ditto, which is the SOTA EM method based on pretrained language models [Li et al., 2020]. We reproduce and obtain comparable results on existing benchmarks.

3 Benchmark Reconstruction for EM

Applications of entity matching in the open environment are highly diversified and complicated. So it is difficult to build one benchmark for all downstream EM applications. To this end, this paper investigates the three most representative applications of entity matching and evaluates how well the current SOTA approaches can accomplish these applications. Specifically, we build benchmarks for the following three kinds of typical downstream paradigms of entity matching:

• Open Matching (OM), which tries to identify whether two open records refer to the same entity clusters, and there is no specific restriction for the entity clusters. This paradigm corresponds to the scenarios where we want to leverage the learned entity matcher as a universal tool for merging or deduplicating (e.g., combining two databases, deduplicating a new table, etc.). As a result, the clusters and records in the test benchmark are commonly unknown during training under this scenario.
• Cluster-focusedMatching (CFM), which tries to identify whether two records belonging to specific seen clusters are matched. This paradigm corresponds to the scenarios where we try to sort the mined records about a group of products or merge information about specific

entities from multiple sources. Under this scenario, all entity clusters in the test set are observed during training, but all records in the test benchmark are new and unseen.

• Record Linking (RL), which tries to link a newlyobtained record to other records in the database. Under this scenario, we can use all entity clusters in the database for training, and there is no new cluster in the test set. However, for each record pair to classify, only one record in it has appeared in the training set, and another record is new. This is a very common scenario for entity matching applications, e.g., linking a new product record to the constructed commodity library.

Apart from the above three benchmarks, we also build a vanilla benchmark for comparison, which follows the construction criteria of the WDC benchmarks [Primpeli et al., 2019]. Figure 2 illustrates how we build these benchmarks upon our constructed corpus. All these 4 benchmarks share the same training and validation sets. For the test benchmark of OM, we apply the same record pair generation procedure as Algorithm 1 on the hold-out clusters to generate matched and mismatched record pairs. For CFM, we use the hold-out records from the training set to build the test benchmark. For RL, we first sample a record from the hold-out records and then sample a matched or mismatched record to it from the training set. We set the default matched-mismatched ratio on training and test sets to 1:3 like the WDC 1 . However, in experiments, we also vary the matched-mismatched ratio of the benchmarks to investigate how the balanced label assumption influences EM performance. Besides, following the same criteria, we also build these 4 benchmarks for each category.

4 Experiments and Findings

4.1 Restricted Entity Assumption

Findings 1. Restricted entity assumption biases the nature of entity matching, which changes the task from learning an effective matcher to learning effective representations of seen clusters/records.

To demonstrate this, we compared the performance of the SOTA entity matchers on 4 newly constructed benchmarks. Table 2 shows the results. We can find that on the vanilla benchmark, these methods can achieve competitive performance of nearly 90% F1 scores, which is similar to that on previous benchmarks. Unfortunately, the performance dramatically dropped when the restricted entity assumption was removed. We find that the model performance dropped most significantly in the realistic Open Matching scenario - nearly 30% of F1 scores. Meanwhile, the performance also have a significant drop in Record Linking and Cluster-focused Matching, even most clusters or records in these two settings have been observed during training. This verifies that previous benchmarks incorporated with restricted entity assumption significantly overestimate the performance. Furthermore, we find that the more information about seen records or clusters in benchmarks, the better the performance of the models. This shows that current models, trained on standard benchmarks, do not learn sufficient information to build a universal matcher. Instead, they pay more attention to learning sufficient representations of seen clusters and records. However, in order for the entity matcher can be used in a broader range of scenarios, we hope that the learned entity matcher can be generalized to unseen clusters and records. As a result, only focusing on learning representations of seen clusters or records is not sufficient for EM in an open environment.

1 webdatacommons.org/largescaleproductcorpus/v2/#toc4.2

In general, previous benchmarks mainly evaluate the ability to learn on seen records or clusters but cannot reflect the ability of generalized entity matching in the open environment. Therefore, there is a discrepancy between previous benchmarks and the real-world entity matching applications.

4.2 Balanced Label Assumption

Findings 2. Balanced label assumption conceals the most critical challenge of entity matching.

To show this, we investigated the performance of the baselines on 4 newly constructed benchmarks and varied the ratio of matched record pairs to mismatched record pairs gradually from 1:3 to 1:100. Note that 1:100 is a reasonable matchedmismatched ratio in real-world entity matching applications, because the entity resolution system may keep more than 100 candidates for a record at the blocking step to ensure recall [Thirumuruganathan et al., 2021]. And due to the longtail phenomenon, it is very frequently that only one instance among them is the matched pair. So a matched-mismatched ratio of 1:100 corresponds to real-world EM applications.

Table 3 shows the results. We can see that even on the vanilla benchmark, the F1 score dropped dramatically on the imbalanced scenario, compared to the results on the balanced scenario in Table 2. More importantly, the performance becomes even worse in other three benchmarks. On the open matching benchmark, the performance of state-of-the-art systems is even as low as 14.52%. This shows the significant impact of imbalanced labels on entity matching.

In order to take a closer look at the challenges posed by imbalanced labels, we dynamically varied the mismatchedmatched ratio on test benchmark. Figure 3 shows the results. We can see that the performance on all benchmarks steadily dropped when the mismatched-matched ratio increased on the test benchmark. Furthermore, we find that this problem can not be solved by merely adjusting the mismatched-matched ratio on the training set, as shown in Figure 4. We believe that this is because the imbalanced training set will pose a great challenge for model learning. As a result, how to deal with the extremely imbalanced labels in the open environment is one of the most critical challenges for entity matching. Unfortunately, previous benchmarks conceal this challenge because they introduced the balanced label assumption when generating mismatched instances. Consequently, previous benchmarks can not well represent the real-world performance of entity matchers in an open environment.

4.3 Single Modality Assumption

Findings 3. Single modality assumption stems from the underestimation of the importance of multi-modality on previous benchmarks.

To show this, we conducted experiments on newly constructed benchmarks with multi-modal records. Because currently there is little previous work focused on multi-modal entity matching, we build a simple baseline for visual and multimodal EM. Specifically, we use Vision Transformer [Dosovitskiy et al., 2021] as the image encoder and apply a multilayer perceptron on the representations of images of two records to obtain a visual matching representation. Then for the singlemodal visual baseline, we direct send the visual matching representation into a classifier to determine the match result. For the multi-modal approach, we use a gated mechanism to fuse this visual matching representation with the text matching representation and then send it to a classifier.

Table 4 shows the results. We can find that the importance of visual attributes can be underestimated based on the performance of the vanilla benchmarks. In balanced vanilla benchmarks, the improvement of introducing visual information is not very significant, which is consistent with the phenomenon from previous benchmarks [Wilke and Rahm, 2021]. However, when we refer to the results on open environment benchmarks, the improvement of incorporating visual information is very significant: there are more than 7 points of F1 score gains on CFM and more than 11 points of F1 score gains on OM. Furthermore, the multi-modal model achieves more than 40% of F1 scores improvements under the imbalanced benchmarks for some categories and can be better generalized to unseen clusters and records. All these results demonstrate that multi-modal information can significantly benefit entity matching in the open environment.

5 RelatedWork

EMApproaches. Entitymatching aims to identify whether two entity records refer to the same real-world entity, which is the most critical step of entity resolution [Christophides et al., 2021]. This study dates back to [Fellegi and Sunter, 1969] and has attached great attention. To solve this open problem, various approaches have been proposed, e.g., distance-based, rule-based, declarative, and probabilistic methods [Papadakis et al., 2021]. In recent years, deep learning has been introduced to this field and achieved promising results [Thirumuruganathan et al., 2018; Mudgal et al., 2018; Nie et al., 2019; Zhao and He, 2019; Fu et al., 2020; Shao et al., 2020; Li et al., 2020; Ge et al., 2021].

EM Benchmarks. In the early development of EM, many datasets are used to construct benchmarks to evaluate EM methods [Primpeli and Bizer, 2020]. There has been an effort on building large-scale datasets for deep learning methods [Primpeli et al., 2019]. There have also been some attempts on extending the EM task to broader scenarios by extending the data schema, record formats, and relationships between records [Jim´enez-Ruiz et al., 2020; Wang et al., 2021]. Unfortunately, as we mentioned above, all these benchmarks are built on three erroneous assumptions, which lead to a significant gap between the benchmarks and EM in the open environment. There is also some literature discussing multi-modal entity matching [Christophides et al., 2021; Wilke and Rahm, 2021]. However, due to the benchmark limitation, the importance and effectiveness of multi-modal attributes to EM were hindered and inaccurately evaluated.

6 Conclusions

In this paper, we highlight that the gap between reality and ideality of entity matching stems from the erroneous implicit assumptions introduced during the benchmark construction process. These assumptions are inconsistent with the nature of entity matching and therefore lead to biased evaluations of current EM approaches. To this end, we build a new EM corpus and re-construct EM benchmarks. By step-wisely changing the restricted entities, balanced labels, and singlemodal records in previous benchmarks into open entities, imbalanced labels, and multi-modal records in an open environment, we find that current state-of-the-art approaches suffer severely from unseen clusters, imbalanced labels. Furthermore, previous benchmarks also underestimated the impact of multi-modal attributes on entity matching. Our findings reveal that previous benchmarks biased the evaluation of the progress of current entity matching approaches, and there is still a long way to go to build effective entity matchers.

Acknowledgments

This work was supported by the National Key Research and Development Program of China (No. 2020AAA0106400), and the National Natural Science Foundation of China under Grants no. 62122077, 62106251, 62076233.