![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
How Many Times Must a Tech Provider Download the Same Listings?
I received a call recently from an MLS administrator who wanted to talk about a RETS issue that had been bothering him. His MLS charges a small fee to subscribers for a RETS feed; the fee covers the costs related to the feed, including compliance audits. He was noticing that many of the RETS credentials that subscribers were paying for weren’t being used and thought this was a bit of a mystery. Should he disable the unused RETS credentials and stop charging the subscribers? That course of action would make sense if his subscribers truly no longer needed the data. But there was a more likely culprit behind most of his mystery.
Quite often a subscriber’s RETS feed isn’t just associated with the subscriber, but with a third-party vendor providing IDX, VOW, CMA, statistics, and/or broker back-office systems to multiple MLS subscribers. Let’s say the vendor has downloaded the IDX data on behalf of one broker. If the vendor has 19 more customers associated with that MLS, does it really make sense for the vendor to download and store the data 19 more times, using the additional 19 RETS credentials? That seems like a real waste of server, bandwidth, and storage resources. On the other hand, suppose the vendor re-uses the credentials. Further suppose that the MLS administrator turns off unused credentials, the subscriber whose credentials have been used by the vendor to download data goes inactive, and his or her credentials are disabled by the MLS. The flow of data to the other 19 websites will be cut off. That’s not good!
Besides the potential for data disruption, there are other reasons why an MLS administrator may not like re-use of credentials:
1. Credential re-use takes authorization control out of the hands of the MLS. If the vendor doesn’t know that a subscriber whose credentials they aren’t using has gone inactive, the vendor may accidentally service him or her using data obtained using another subscriber’s credentials.
2. Similarly, re-use may defeat opt-outs for individual uses.
3. The problem is actually even more complex if the vendor has multiple products. The vendor may download a superset of all data they need for a broker back-office use. Then, by re-using a subset of the data for an IDX site, the vendor may accidentally use fields and listings in certain statuses that would not normally be available to the IDX feed, inadvertently using the data inappropriately.
4. Credential re-use partially defeats the use of data seeding, i.e., trying to figure out where exactly there’s a data leak.
Having unpacked some of the issues, there seem to actually be two questions regarding RETS credential re-use that need to be considered:
1. Is it okay to re-use data feed credentials for multiple parties with the same use?
2. Is it okay to re-use data feed credentials for one or more parties with different uses?
So, re-stating the conundrum simply: it’s terribly inefficient for all parties when vendors download and store multiple copies of data, one for each customer and credential, but there are valid reasons why MLSs have looked negatively at the practice of credential re-use. How do we solve this for everyone?
There are several possible ways to address the authorization control and opt-out issues including, but surely not limited to, the following:
The vendor can log in using all MLS-provided credentials at least once per day to figure out what subscribers no longer have rights to use data based on RETS login failure. They won’t download data with each login, just for one of them. But this way, the MLS will have a record that the vendor has checked whether a login / use is still active on the RETS server and should have taken steps to eliminate data use for that subscriber.
The vendor can be given a RETS login by the MLS that gives the vendor access to the roster, limited to a subscriber identifier and status (active, inactive). The vendor can use this to check if they need to stop re-using credentials on behalf of a specific customer.
RETS standard and server functions can be designed to return validation codes for all authorized specific MLS users and uses based on a single login credential, and return data based on that information. This will directly reflect the kind of master agreements and addendums that many MLSs have with these vendors already. If no MLS users are active and related to a vendor credential, the vendor credential will not provide data access.
The inappropriate data use issue is a bit trickier. It is an issue that can be mitigated today to some degree via very clear license agreements, vendors being careful to use the data subsets as specified by those agreements, and by MLSs auditing the end-uses of the data (i.e., the IDX websites and VOWs) – something they should be doing anyway. Additional mitigations may require some RETS standard and server-side function enhancements. For example, additional usage opting information can be passed to vendors where relevant. Also, a server-side function could be created to efficiently determine whether several credentials provide different data for a query – without downloading and comparing the data to the data on the client side. Knowing that different credential use would provide different data may make it easier for a vendor to know whether re-use is appropriate or not.
I don’t think there’s a way to fully resolve issue the data seeding issue while allowing credential re-use but tracking an issue down to who received the feed is still possible. Vendors just need to cooperate with any seeding investigation to help figure out what specific usage is involved. Data seeding is only of use in a very limited subset of illegitimate use detections anyway.
There are more conversations to have on this subject, looking at additional business and legal issues as well as technical reflections of those issues, but this is a starting point. Let’s figure this out, so that RETS service can be efficiently provided to stakeholders while addressing legitimate issues that arise with that efficiency. What’s next? Let’s discuss these and other ideas for solving the issue here on this blog, on Facebook, and perhaps at the upcoming RESO meeting and see if some consensus can be reached among both vendors and MLSs. If changes to RETS are desired, this can be dealt with in RESO workgroups and implemented by vendors as need be.
I know many vendors that simply must engage in credential re-use so they don’t overwhelm MLS RETS servers and so they don’t needlessly increase their costs to service multiple customers – but they don’t like being in violation of some of their license agreements with regard to credential use. I’ve even had clients fine such vendors – and while this is in accord with the letter of some current license agreements, it’s really not fair. These are not “bad vendors.” By not defining our standards, process and legal agreements to reflect the technical reality of data aggregation and use, we’ve created this ugly issue together. But together, we can solve it, and we should do so as quickly as possible.
I received a call recently from an MLS administrator who wanted to talk about a RETS issue that had been bothering him. His MLS charges a small fee to subscribers for a RETS feed; the fee covers the costs related to the feed, including compliance audits. He was noticing that many of the RETS credentials that subscribers were paying for weren’t being used and thought this was a bit of a mystery. Should he disable the unused RETS credentials and stop charging the subscribers? That course of action would make sense if his subscribers truly no longer needed the data. But there was a more likely culprit behind most of his mystery.
Quite often a subscriber’s RETS feed isn’t just associated with the subscriber, but with a third-party vendor providing IDX, VOW, CMA, statistics, and/or broker back-office systems to multiple MLS subscribers. Let’s say the vendor has downloaded the IDX data on behalf of one broker. If the vendor has 19 more customers associated with that MLS, does it really make sense for the vendor to download and store the data 19 more times, using the additional 19 RETS credentials? That seems like a real waste of server, bandwidth, and storage resources. On the other hand, suppose the vendor re-uses the credentials. Further suppose that the MLS administrator turns off unused credentials, the subscriber whose credentials have been used by the vendor to download data goes inactive, and his or her credentials are disabled by the MLS. The flow of data to the other 19 websites will be cut off. That’s not good!
Besides the potential for data disruption, there are other reasons why an MLS administrator may not like re-use of credentials:
1. Credential re-use takes authorization control out of the hands of the MLS. If the vendor doesn’t know that a subscriber whose credentials they aren’t using has gone inactive, the vendor may accidentally service him or her using data obtained using another subscriber’s credentials.
2. Similarly, re-use may defeat opt-outs for individual uses.
3. The problem is actually even more complex if the vendor has multiple products. The vendor may download a superset of all data they need for a broker back-office use. Then, by re-using a subset of the data for an IDX site, the vendor may accidentally use fields and listings in certain statuses that would not normally be available to the IDX feed, inadvertently using the data inappropriately.
4. Credential re-use partially defeats the use of data seeding, i.e., trying to figure out where exactly there’s a data leak.
Having unpacked some of the issues, there seem to actually be two questions regarding RETS credential re-use that need to be considered:
1. Is it okay to re-use data feed credentials for multiple parties with the same use?
2. Is it okay to re-use data feed credentials for one or more parties with different uses?
So, re-stating the conundrum simply: it’s terribly inefficient for all parties when vendors download and store multiple copies of data, one for each customer and credential, but there are valid reasons why MLSs have looked negatively at the practice of credential re-use. How do we solve this for everyone?
There are several possible ways to address the authorization control and opt-out issues including, but surely not limited to, the following:
The vendor can log in using all MLS-provided credentials at least once per day to figure out what subscribers no longer have rights to use data based on RETS login failure. They won’t download data with each login, just for one of them. But this way, the MLS will have a record that the vendor has checked whether a login / use is still active on the RETS server and should have taken steps to eliminate data use for that subscriber.
The vendor can be given a RETS login by the MLS that gives the vendor access to the roster, limited to a subscriber identifier and status (active, inactive). The vendor can use this to check if they need to stop re-using credentials on behalf of a specific customer.
RETS standard and server functions can be designed to return validation codes for all authorized specific MLS users and uses based on a single login credential, and return data based on that information. This will directly reflect the kind of master agreements and addendums that many MLSs have with these vendors already. If no MLS users are active and related to a vendor credential, the vendor credential will not provide data access.
The inappropriate data use issue is a bit trickier. It is an issue that can be mitigated today to some degree via very clear license agreements, vendors being careful to use the data subsets as specified by those agreements, and by MLSs auditing the end-uses of the data (i.e., the IDX websites and VOWs) – something they should be doing anyway. Additional mitigations may require some RETS standard and server-side function enhancements. For example, additional usage opting information can be passed to vendors where relevant. Also, a server-side function could be created to efficiently determine whether several credentials provide different data for a query – without downloading and comparing the data to the data on the client side. Knowing that different credential use would provide different data may make it easier for a vendor to know whether re-use is appropriate or not.
I don’t think there’s a way to fully resolve issue the data seeding issue while allowing credential re-use but tracking an issue down to who received the feed is still possible. Vendors just need to cooperate with any seeding investigation to help figure out what specific usage is involved. Data seeding is only of use in a very limited subset of illegitimate use detections anyway.
There are more conversations to have on this subject, looking at additional business and legal issues as well as technical reflections of those issues, but this is a starting point. Let’s figure this out, so that RETS service can be efficiently provided to stakeholders while addressing legitimate issues that arise with that efficiency. What’s next? Let’s discuss these and other ideas for solving the issue here on this blog, on Facebook, and perhaps at the upcoming RESO meeting and see if some consensus can be reached among both vendors and MLSs. If changes to RETS are desired, this can be dealt with in RESO workgroups and implemented by vendors as need be.
I know many vendors that simply must engage in credential re-use so they don’t overwhelm MLS RETS servers and so they don’t needlessly increase their costs to service multiple customers – but they don’t like being in violation of some of their license agreements with regard to credential use. I’ve even had clients fine such vendors – and while this is in accord with the letter of some current license agreements, it’s really not fair. These are not “bad vendors.” By not defining our standards, process and legal agreements to reflect the technical reality of data aggregation and use, we’ve created this ugly issue together. But together, we can solve it, and we should do so as quickly as possible.