SECSAC Meeting
Real-Time Captioning
15 October 2003

Note: The following is the output of the real-time captioning taken during the Security and Stability Advisory Committee meeting 7 October 2003 in Washington, DC. Although the captioning output is largely accurate, in some cases it is incomplete or inaccurate due to inaudible passages or transcription errors. It is posted as an aid to understanding the proceedings at the session, but should not be treated as an authoritative record.

Steve Crocker: Let me welcome everybody. My name is Steve Crocker. I chair ICANN's Security and Stability Advisory Committee. This is the second of two meetings that we have – that we are holding.

I trust that everybody who has come knows why they're here. We're in a very nice facility at the Academy for Educational Development up here on the eighth floor and the path here is tortuous enough that it's its own test, I think. And I appreciate everybody coming and spending yet more time with us. We met a week and a day ago, and had an extended session with presentations from a number of people looking principally at effects with an introductory presentation from Verisign on what they had introduced in the middle of September.

This time we're going to focus primarily on an extended set of presentations from Verisign on not only what they implemented, but the processes that they used in advance and the information that they've acquired since then. We also are very pleased to have two other presentations: one from Ben Edelman at Harvard and one from Global Name Registry which I think will be coming remotely late in the afternoon. I appreciate the effort everybody has put into this, particularly the people who have traveled -- in some cases, traveled again to come here.

Ben, Paul Vixie, I'm sure I'm going to leave people out if I try to do this carefully. And some people, Steven Bellovin called up and is having train trouble and should show up here by and by. Other people's planes didn't comes off. Russ Mundy sends his regrets, and so forth. I'm going to briefly go through a couple of the slides that I used last week just to make sure everybody is on the same page, and introduce a tiny bit of information that we've accumulated, and then turn the floor over to a quite stellar team from Verisign to take us through all the steps here. So just quickly, ICANN's charter covers the Domain Name System and addressing. ICANN strives to achieve coherence, stability and security.

Almost all of the operational details are actually carried out by others. One component of ICANN is the Internet Assigned Numbers Authority that operates within ICANN. Most of the people have been on the committee from the very beginning. This is a committee of outside experts that was chosen for, first of all, their expertise in the Domain Name System and in the addressing system, and, second of all, for their connectivity to operating parts. Everybody on this committee has an ordinary day job. There are no bureaucrats, no attorneys, no lobbyists, or marketing people, for example, on this committee. Everybody has some connection to an actual operational entity. And we also have a fair amount of geographic breadth.

So we have representatives from root servers, from top-level domain registries, from registrars, from address registries, from the security community, and I'm sure I'm leaving out critical components. But a quite distinguished group that has been in place for many months, unrelated to this particular sequence of events. So we haven't shuffled the deck at all.

And I think I've just covered this. Did I leave anybody out? Regional Address Registries are the address registries, the top-level domain include both generic and country code. Our committee is one within the ICANN universe that deals with security and stability. I've listed us as one of the specialist groups, and IANA being the other and there are constituent committees. This is all in the handouts, I'll skip past it.

Within the entire space of folks who worry about different aspects of the operation of the Internet, here's a two-dimensional picture that goes from architecture up through policies and laws, and has as another dimension, the geographic spread. This is illustrative, definitely far, far from complete. So if your favorite organization is left out, hopefully there's a place for that on this chart. The purpose of this is to show that in a limited but very broad away ICANN fits across the geography and multiple layers of this picture. It doesn't actually do the engineering, of course. It doesn't do the architecture, and it doesn't do the law enforcement or policy and laws, but it touches on several layers of protocol and several layers of this hierarchy and covers the entire global structure. In response to the events that we've been talking about, we opened up a comments line, secsac-comments@icann.org, and acquired lots and lots of e-mail. This is the results after taking out the Spam which easily more than doubled the amount of mail. Just one of the chronic ills that plague us all. We had some help from Elias -- I'm going to blow this, Elias Bachrach, and Paul. Is Paul here? Or is he sleeping?

(inaudible).

Steve Crocker: and Mike St. John and Jim Galvin, combing through the mail and trying to assemble the pieces for us.

We received a handful of positive and negative comments and neutral comments, and broke that down into relevant issues, non-relevant issues, and no issues, it was content free. Our goal is to identify specific things that were broken or right or whatever, and some were just emotive. The complaints, we'll document all of this in more detail. This isn't the time and we haven't got all the pieces laid out, but in broad terms, the largest complaints were about Spam filters, about the fact that the change was introduced without lead time, some privacy issues, bandwidth consumption, and a variety of problems with mail bouncing or being delayed. Some secondary complaints about misleading error codes, inability to opt out, and some difficulty in debugging network issues where it just wasn't clear what was going on and sort of added another layer of complexity. Today, as I said, we're going to have a lengthy focus on Verisign. This is a nominal agenda. What we're actually going to do is have a sequence of talks, and then open up the floor for questions. So in all, I think we're looking at approximately two hours, and I think the Verisign folks want about an hour or so of that, and then there will be, as I say, a lengthy set of -- question-and-answer period. As we did last time, we have members of the committee who are participating by phone and web cast. I'll give them the first opportunity to ask questions. I'll go to our committee members who are here physically, back to the phones, and then wide open from the floor. A couple of points. Let's see. I should try to introduce who is here. I should talk about where the bathrooms are, and the protocol for coming to the microphone. Did I miss something, Jim?

Jim Galvin: SECSAC (inaudible) comments for people on the phone who want to --

Steve Crocker: secsac-comments@icann.org.

The other things, let me introduce the members of the committee who are here. Mike Kosters, and Ray Plzak, Ken Silva, and Steven Bellovin will likely come when Amtrak delivers him from AT&T research, and Jim Galvin who is executive support for the committee. And Jim, remind me who is on the phone.

Jim Galvin: actually, we have Doug Barton, Fredericko Neves, Jaap Akkerhuis, KC Claffy, Ram Mohan, and Suzanne Woolf, and Sam Weiler, and Rick Wesson.

Steve Crocker: that was a lengthy set of people. I was going to try to repeat them back but you've exceeded my short-term memory here.

But an equal number, plus, maybe twice as many people as here, so a good number of people including some invited guests. The microphone is here. Please, when question time comes up, please come up, introduce yourself, and before you talk and while you're standing in line, if there's a line, please write down your name. We are transcribing everything. We are absolutely fantastic transcription services here. If you fail to write your name down, you may trip as they catch you trying to leave the microphone and you'll get a second opportunity to write your name down. Lavatories were on your way in, sort of back toward the elevators. There's coffee and soda, I think, outside continuously. Feel free. We'll take a break, but mostly you should feel free to come and go as you wish. There's handouts from last week, handouts from this week, and a bound material from Verisign as well that just arrived.

Any other logistics that I should have covered? So with that, let me turn the floor over to Anthony Renzette, and we'll move -- oh, I'm sorry. Ben is going -- you're going to say an introductory word? So Ben Turner, Vice President of Verisign, will start things off and sort of orchestrate from there. Thank you.

Ben Turner: as Steve said, my name is Ben Turner and I run the name services group of Verisign. and I want to take a couple of minutes and thank ICANN and SECSAC for inviting us to share this information we've gathered over the last several months. We're also very interested, as Steve referred to, as the pieces come together in the SECSAC information collection to receiving that information and taking a look at that.

So what we want to do today is go through, really, four major, five major topic areas. We're going to have Anthony Renzette take everyone through the pretesting we did. We're going to have Scott Hollenbeck review what our technical panel found, we're going to have Matt Larson go through the technical issues and give you our feedback on them. I'm going to go through the consumer feedback on the service. And then Rusty Lewis is going to close it out. So what I'd like to do is have Anthony come up and take you through the pre-launch information. And if at all possible, we'd like to go all the way through the presentation before there are any questions, because I think it will be more complete if we're able to do that.

Anthony Renzette: thank you. So what you see here and what we'll be talking about during this presentation of the larger presentation is the actual activities that we went through prior to developing the actual concept for SiteFinder, prior to developing the product and service that became SiteFinder, as well as the different activities that we did along the way that were extraneous and extra steps we did to ensure the development went smoothly and without the hitch.

The concept of SiteFinder itself came from research we did and in the course of being a registry. In the course of providing our registry services business we routinely conduct research in the market on buying behaviors of domain name holders, people that use domain names, and the types of services that they like to see when purchasing domain names, when purchasing value-added services on top of the domain names and the things they like to see that help them utilize those domain names fully, and then also, of course, renew and continue along the cycle with our customers. The research that we did in this particular case took place in October of 2002. It was designed to, again, assess the needs of consumers and small businesses known as small or home office (SOHO's). It included over 1300 respondents, conducted in on-line interviews representing different demographic areas, and the confidence in this research was a 95% confidence level with an error level of plus or minus 3%. What's interesting about this research is, as I said, the majority of the questions and the research in it are designed to determine buying behavior and likeliness to renew and attitudes and products that affect that. We also asked open-ended questions at the end of this research to assess some of the current pains the Internet users and this type of user have in particular. Of the things that came back in this, one of the top three items that came back was, quote, new ways to find URLs you're attempting to find which drilled down to spell correction on the web. This was the first part that let us on the road to SiteFinder. .

Having this initial concept we of course need to do more research. We realize, like many product development companies, that having initial concept isn't enough. You need to flesh it out, you need to ask the market what they mean by these concepts. More importantly, you need to figure out what types of services would be proven beneficial to the end customer. To do this, in this particular case we conducted two very identical but with different respondent sets, but two identical in methodology sets of research. One in December of 2002, so very soon after getting that introductory feedback, and the second in January of this year. In this case, we floated out the idea, we compared the concept of a spell correction page and response set to the additional error page that people already see. What was very positive is that there was a very high level of awareness. People knew what we were speaking about when we talked about the traditional error response page.

We then floated out a number of different features that could be included on a new response page, and there are two that floated very highly to the top. The first was an ability to initiate search. You can see in December 2002, approximately two-thirds or 67% of users thought that would be highly beneficial, and also links to related or relevant sites accounted for 65% highly useful criteria. In January of 2002, we are encouraged to find that these users now thought that the preference towards search had risen to 70% while links to related sites had risen even higher to 68%. The final take-away from this was that spell correction was, indeed, a feature that users liked, but we also need to include two additional features: of course, the ability to initiate search, and links to related sites and capabilities. So how did we get from this concept to then what we know today as the SiteFinder service? Well, in developing any product or service, we always try to follow a couple of key rules to live by. These are all equally weighted and of equal importance, the first of which is to meet the end-user demand; in this case, to improve web browsing experience as you saw from the original concepts. Second, of course, is any service that we launch, strive fully to be standards compliant. Also in a business of our nature and our size and impact, the service must be scalable. And of course, the service must be able to maintain stability and security of the Internet in general. We, like many other companies, were aware of wildcards that existed out in other registries, we found two that utilize them today in dot tv and dot cc. And we also are knowledgeable of other registries that are using similar work. What we really did here that was new for us, and I think new for the SiteFinder product, was that we took this concept, this spell correction, we added it to a concept to a wildcard in the zone and came up with this, the wildcard was returned when someone mistyped the name, it would include a correction of what the user had mistyped to get them there, as well as the search box functionality and links to related web sites. Of course, as I just mentioned, there are a number of rules that we live by.

Meeting end-user demand was one of them. We also need to take seriously these issues of scale and stability. One of the first things that we need to do is to try to figure out exactly what does that mean, what is the volume, what type of traffic would we see with this service?

Obviously doing the number of transactions that we do, this could be a very large number. The way we do this is that we actually contracted with an external group to provide some statistical help and using that, pulled 30 random samples over a period of seven days. Each of these samples included approximately 3,000 responses. This ranged across the entire dot com, dot net system, which yielded a total of 16 million responses collected. We had a statistician helping us with the analogy, that a 95% confidence level would yield us a plus or minus error response of 5%. We got a really detailed review of what the DNS traffic would like with such a service. Of the approximately 300 billion monthly DNS requests that we receive and process, we would see approximately 600 million monthly name error responses resulting from web browsers. Very important for determining how the system should be built. This also provided insight into the types of requests which we would drill upon further, and it gave us an indication of how the system overall should be built. An interesting take-away from this is, now, looking back on it, that pre-launch analysis is very, very similar to the data that we received during the actual operation of the SiteFinder service. We also then went to another external group to try to get an idea of what the impact would be, the effects would be of a wildcard response to request for nonexistence domains across various applications. This external party was a group of external DNS experts brought in to look across 13 different categories. You can see some of the examples there. 37 different protocols, and 53 different implementations of those protocols, common end user applications. The results of this testing and analysis, which was fairly detailed, produced a recommended course of action on how we should introduce this.

One of the items that they brought to our attention and recommended strongly was the implementation of a stub server in the instance of e-mail, which of course we followed, and also implemented what they termed in their response a, quote, TCP reset option. what they specifically meant by that was requests for non-HTTP and non-SMTP traffic be responded to as follows in the case of TCP, receive connection refused response, UDP, ICMP, port unreachable. Their conclusion doing that analysis and research is that the user response would not change dramatically given this implementation.

This, of course, was not enough for us to be able to proceed. We also needed to do some controlled live testing. In this testing what we were doing was actually giving a wildcard response and a record response instead of a name error in certain very brief and small instances. In this particular case we ran a total of 12 minutes of testing in four different types, receiving a total of 61,000 wildcard responses which yielded 194,000, give or take, hits at our response server that we had set up at the time. You can see that the ratio of response over hits per wildcard response is approximately 3.16 and that had varied somewhere between 3.16 and 5.5. The critical piece of this is it helped us validate some of the earlier research we had done on the protocol analysis, what the spread of those protocols would be, as well as confirming our assumptions on the size and capacity a service of this nature would require. Of course before launching any service of this nature it's also good to get field testing. In this case we were able to work with a diverse range of companies, these were not Verisign subsidiaries. We contacted them through a very complex and externally assisted range of survey and reviews, asking various questions about their reliance on DNS, existing Internet behavior and so forth.

The idea behind this was to identify what responses their production systems would have to a SiteFinder solution.

During the course of this, we contacted over 600 companies. 55 of those companies decided that they wanted to get the full briefing that was interaction with our operations teams, answered questions, and so forth.

And of those 55, 35 decided that this was significant that they need todayed to participated in the testing themselves. This ranged across several industries which was health care, transportation sector, telecom, web crawlers, financial, and software companies. The companies that conducted this testing did it in a number of different ways. One, they used their QA environments and some even used production systems to test these applications against a DNS response server which is configured to give a wildcard response. So their systems in this system would be receiving a wildcard response to a nondomain name query. They also had custom applications and tested custom applications. They tested key applications and some were set to intentionally misconfigure key domain names within their systems to see how the response would be determined. The testing of this nature of course must be based on the responses we received from those companies. We obviously rely strongly on their responses, and the responses we saw from all of them was that they had no major issues to report. There were no impacts on their operations and that the wildcard response did not impact their software applications or their systems.

When developing the product further, so we've progressed now from the concept that it was taking it through the end-user research, taking it further through the analysis on how the service would be responded to, and now we're actually in the product development phase. We understood that the impacts and the responses received would be varied, and that wherever possible, documentation on the methodology followed would be extremely important. This set of documentation is of course available and has been available on the SiteFinder and Verisign sites for quite some time, but specifically, we have a white paper that discusses the implementation of DNS wildcards in general. In specific, we have an implementation guide that covers Verisign SiteFinder, how we followed the white paper, the steps that we took. Additionally there's an application developers guide that was developed by our team in conjunction with others looking at how people who develop software in systems that externally rely on DNS could work with a wildcard in a SiteFinder response. We also, of course, use a combination again of external and internal resources when developing the product itself, and more importantly, when testing and reviewing our processes and procedures. This included external QA or quality assurance testing that was brought in, in addition to our own internal teams to make sure that the steps that they followed met and were as objective as possible. And then we also took our product development processes, our QA processes, our pre-launch processes, and those systems and procedures for another review by an external company familiar with software and Internet software development, and made sure that they reviewed how we were doing this, helped us make modifications and ensured completeness. And then in addition, as Scott Hollenbeck has mentioned in a previous statement, the product itself was released with a monitoring program built in so that we could measure any impacts in responses that we saw once it was live. with that, I'd like to turn it over to Scott Hollenbeck to review the technical review panel that we set up.

Scott Hollenbeck: okay. thank you, Anthony, and thank you, Steve.

Thank you for the opportunity to address the committee this afternoon.

As Anthony said, my name is Scott Hollenbeck. I'm here to tell you a little bit about our technical review panel and to describe the outputs of that panel. overview of the presentation in front of you.

First, the purpose of the panel itself. Soon after the launch of SiteFinder, Verisign announced we were going to form a technical review panel of outside experts to specifically solicit and gather technical information and data regarding the implementation of SiteFinder. We were looking primarily to obtain some objective, sane analysis of a lot of the commentary that was being made in various public forums that we were receiving at our customer service department and at various public e-mail addresses we made available. Stage two was to distill the received information and data into specific implementation issues. After we identified the issues, trying to determine which issues were based on fact concerning the service, and which were based on emotional or political considerations. For each issue associated with the service, determine the likelihood of the issue arising for Internet users and then the consequences of the issue for users. And then based on the analysis of the issues, determine what enhancements, if any, could be made to improve the service, or to provide a work-around for people who are impacted. And then finally, the panel took it upon themselves to report the observed implementation issues to Verisign along with any data supporting the issues, and then Verisign providing the final analysis, ran it by the panel for their approval, and you'll actually see the raw data provided by the panel at the end of this presentation.

So who was the panel? We tried to cast a very wide net to collect experts in various industries, specifically some related with Spam detection and prevention, that's Ken Schneider, the CTO and VP of operations at Brightmail, George Sherman is the CTO of Morgan Stanley with a strong background in Information Technology, Keith Teare, Chairman and President and CEO of Santa Cruz Networks. Bruce Tonkin, CTO of Melbourne IT, and we also have three other members of the panel who wished to remain nameless because they felt they could be more effective if they weren't subject to some criticisms from people floating around in the community at the time. We also had four senior Verisign engineers participating on the panel, Leslie Daigle, myself, Mark Kosters and Matt Larson. Bruce set the agenda and we were there to make sure the panel had accurate information from a Verisign perspective.

So what did the panel do? We tried to look at SiteFinder from three different angles: one was an analysis of all the reported issues that we could find from the various sources I described earlier. Two, we tried to look at SiteFinder from analysis of the protocols that were involved in contacting the response server. Three, we tried to look at what people sitting in chairs were actually doing when they experienced or somehow dealt with SiteFinder, the service. And as far as the sources go, we looked at the statement produced by the IAB and issues presented in other forums, such as the NANOG mailing list, Slashdot, the on-line press, and frankly any other place where we could find sane national commentary.

Okay, analysis of issues. This isn't a complete list of the issues. You can find a complete list in the tables provided at the end of the presentation. But what we tried to talk about here today are those that the panel categorized as having at least a moderate impact on a person. And those are as follows: first that SiteFinder currently provides an English-only web page. And the panel's analysis was that this was an issue that could be addressed by the service operator. End-user error reporting. The panel felt that this was an issue that required software update on some person's part, or some company's part. Spam filtering could be addressed but it required a filter update to do so. Automated HTTP tools could be addressed via a software update. Resolvers that have non-DNS fallback mechanisms such as on Microsoft platforms where if DNS fails you might do a NetBIOS query. Again, the panel felt this was an issue that could be resolved with a software update. Another moderate impact area was the way some people checked domain name availability for registration capabilities. Again, moderate because DNS would be required to change the existing behavior. And finally, issues associated with e-mail delivery, most of which could be addressed by the service operator. From a protocol analysis perspective, the panel looked specifically at the top ten protocols by number of connection attempts. now, this table -- I'm sorry, this pie chart is actually simply a graphic representation of the table I presented at the last review session. As you can see, HTTP and SMTP traffic account for 85% of the traffic we see at SiteFinder.

The panel felt the impact on other protocols was typically a different error message the user would see and/or a slight delay that would be experienced when compared to the pre-SiteFinder experience. The most significant issue from the panel's perspective was that applications don't treat TCP and UDP errors the same way they treat DNS name errors.

So what did we take out of all of this? First, that SiteFinder did not introduce any catastrophic problems. Virtually all of the issues that the panel looked at were categorized as having a minor impact with an unlikely possibility of occurring, but I need to say that that does not mean that they could not occur or that there was no impact. The panel specifically did not identify any issues that had a security or stability issue. The panel did stress the desirability of providing time to adapt and to educate for issues that can't be addressed by the TLD operator. Most of the issues were deemed to be minor or an inconvenience. However, there were some that I described earlier, issues that were categorized as moderate because they would require software change or some proactive change by people outside of the TLD operator. And the next couple of slides, which I won't go into in detail, there's just far too many words here and too difficult to see from the audience but these are the raw outputs from the panel that Verisign's analysis of the discussion that took place. So what you see here is the protocol analysis table, the application use-case table, and the issues table. And in some places where you see empty cells, that's because panel couldn't reach agreement on what the issue was or what the mitigation strategy might be.

That's everything I had. Thank you again for the time, and I'd like to introduce Matt Larson who will actually take you through the discussion of the specific issues that the panel deemed to have moderate impacts.

Matt Larson: hello everyone, thank you, Scott.

So, as Scott said what I'm going to do is walk through what were in the TRP's opinion -- the technical review panel's opinion the most significant issues. And for each issue what I'd like to do is identify and offer a little more explanation than Scott was able to do in the time that he had. Then I'd like to present some applicable data that we have and then finally provide our analysis and our response. So, once again these are the issues, this is the same list that you saw a few moments ago in Scott's presentation.

So the first of these is the issue that the SiteFinder response page was available only in English. And the is that while this is only English previously prior to SiteFinder the browser's error page would potentially have been localized and available in a language understandable by the user. So, Verisign has always planned to introduce a localized version of SiteFinder that just didn't happen with the initial release. So, future releases will include support for at least German, Japanese, Spanish, French, Chinese and other language as our analysis shows that that is what people who are using the site would prefer. The plan is to use the HTTP accept language header display the language and also what many sites, websites allow users to change the language once the -- users to allow the page to display if they're not able to understand the language.

The second of these issues is end-user error reporting. The issue here is that the application behavior in the case of failure changes. And we believe this is at base a user interface issues, the application still fails but potentially with the different error message to the user. So prior to SiteFinder, prior to the wildcard a record and dot com and dot net user attempting a record with application would that have seen. But prior to SiteFinder they would that application would attempt to make a connection and then would see some sort of connection. Behavior change for the users. So, to put in this connection there's no change in application behavior for for domain names. any computer using domain names that work prior to SiteFinder and wildcard also worked afterward. I guess the -- I think the take away here is that a failure is still a failure although we do acknowledge that there's potentially increased user confusion and increased difficulty troubleshooting. For future applications it is possible that application could check for the presence of a wildcard a record on a zone, it could then record to detect synthesized responses and take appropriate action and display appropriate message. I guess one possibility that we have sort of been discussing internally and would like to begin talking about in the community is the potential for a DNS protocol change to indicate a synthesized response. So that a resolver could understand that the response it received was the result of a wildcard and pass that to a wildcard and pass that back to application.

This is a fundamental user interface doesn't affect stability of security. One issue that has gotten a lot of attention it was identified by the technical review panel that was Spam filtering based on domain name existence checks was affected by the existence of the wildcard. And our analysis and reports that we've received from third parties indicate that this issue is more complicated than it might appear at first glance, and that it's perhaps less significant than at sometimes been reported. So, I guess we feel that the reality is that using domain existence checking to identify Spam is first off slow and resource intensive. There's at least one popular mail transport agent whose author believes that this check is not even worth implementing. Feels quite strongly about it.

This economic is also not the obvious straight forward test that it might appear to be. I have more on that in a subsequent slide.

It's also not effective based on our research against a large percentage of Spam. and ideally which I think has gotten lost in some of the discussion of this it really is only one test. Total anti-Spam solution.

I think anyone who is even had a passing acquaintance with the problem of filtering Spam realizes that there's no magic bullet, there's no single test that's going to prevent all Spam. And to that end this is really only the first of possible checks, this is one low-level check.

So, just to be clear, the amount of Spam on the Internet did not increase as a result of the presentation of the wildcard. Let me tell you a little bit about our analysis. We took a look at a very large corpus of Spam. It was over 73,000 messages. And only 3% of those messages contained a nonexistent domain name in the from header. This check can be done different ways and means different things to different people. The first time we went through we did this using an NS query against the dot com and dot net servers. We looked at the domain name in the corpus header. We pared that down to something dot com and something dot net. We did a query to that's where our 3% number comes from. One of the members of the technical review panel took a look at SpamAssassin -- it's Bayesian filters one of the large corpus that is 70% Spam and 30% legitimate mail. On the slide the stats that resulted from that from checking using SpamAssassin's particular check for nonexistence that number is slightly higher, 4.6% of all the Spam in the corpus had a nonexistent domain in the from header. That's this where -- no where near the double digits reported. We just have a hard time reconciling the analysis that we've done in red with some of the other reports that we've seen on this particular issue. So as I mentioned a moment ago, what this has made us realize in the process of doing our own testing is that the whole idea of doing domain existence checking for Spam filtering is a whole -- very subtle process. There's no standard way to do this. And implementations vary in how they do this check. So, for example, gethostbyname in the byname library is not the best way to do this. It only queries for A or quad-A records. In looking at the code for some implementations what they have seen is some Spam filters use this method then don't differentiate between rcode 3, which is the name error or a NXDOMAIN error and no data or no such data error. And what this means is, if an implementation checks for domain existence in this way, they would not be able to tell a domain that didn't exist versus one that has an MX record which could receive mail but not an A record so gethostbyname could have failed.

Unless the application goes out of its way to check the return value from gethostbyname carefully I think what we found is that it's very easy to have false positives to reject mail, that really could be legitimate but this initial filter would cause it to be flagged as Spam.

So, having discovered this we took one of those Spam corpuses and ran it through looking only -- ran it through the same checks, this time using gethostbyname, and purposefully only use gethostbyname at a high level, didn't return success was the structure populated or did we get a null structure, we didn't check the response code and in this case we found that 17% of the Spam messages were identified. So, we think this could be the source of some of the different numbers that we're seeing. That there could be false positives as I said a moment ago. Not in this case right here, because we were looking at Spam exclusively but if you're using this method it's quite possible since there's certainly cases where a domain name has an MX record but no A record, that you would use this test using gethostbyname in this manner and identify it as Spam.

All that being said, I think another thing that's important to point out is that there are people who do feel that this is a check that they want to have as part of their anti-Spam arsenal. and to that end what we saw is that within days and sometimes within even hours of the addition of the wildcard A record in dot com and dot net in various mail transport agents that would issue patches that would allow them to do this check with the presence of the wildcard A record in dot com and dot net.

Another issue was various automated process, in other words not human at a web browser using HTTP over TCP port 80 that think could exhibit problems when encountering the SiteFinder page. Instead of nonexistent domain in causing abort connection couldn't take place, they would attempt to connect to the SiteFinder page, see the SiteFinder response page and then potentially be confused. We have not heard any reported occurrences of this, so we're not able to offer any specific data. No one has told us of any application that was in this situation. In the event of the process that specifically crawl the web, web crawlers and other spidering tools like that, the site does include a robots.txt file which would prevent them indexing the site. I guess more basically using HTTP over port 80 in this way by automated tool is discouraged according to BCP 56. The next issue identified by the technical review panel was name resolution process that continue with other methods, if DNS fails. So Scott mentioned this briefly a moment ago, some versions of Microsoft operating systems and may very well be others that could be configured in this way, if they use multiple methods to do name resolutions, such as DNS and another method like NetBIOS and if they're configured to use DNS first and if they're configured to stop when they get a successful response it's possible that the presence of the wildcard would then cause them to believe that name resolution had succeeded they wouldn't continue on with another method such as NetBIOS.

So, in some cases case a workaround is available, some cases revolve changes the order in which the methods are used and use DNS methods last. We are aware of some configurations are using and intentionally to force resolution to the next method. So, for example, using such a name in a resolve or search list that essentially forced every name resolution attempt to try DNS, to fail because of the intentionally nonexistent domain then proceed onward to the next method. and building configuration that relies on the nonexistence of a domain name like this, is really very dangerous and unwise. Particularly given in the case of dot com and dot net any one can register any domain name they like. That name could be registered out from under someone. And if anyone has looked at the density of names in the dot com and dot net zones it's amazing the kinds of names that are registered, it can be nontrivial to pick a nonexistent domain even one that you would think ought not to be there really, really is. And additionally RFC 2606 does define top level domains and sample dot com and dot net domain names.

For example, it could be used to construct nonexistent domain name that could be used for this purpose. Some applications use a DNS address record query to check for domain name availability and they would potentially be affected by the presence of the wildcard a record. Now, I think it's important to point out that there are various names that don't get put in the dot com and dot net. For example, reserve names, names that are on hold, and domain names that are registered but don't have any names specified. None of these mix it into the zone, there for any kind of a DNS query for them would not detect them they would be potentially viewed as available when in reality they're registered and unavailable. So using DNS for this purpose is not recommended.

Registrars that want to perform this check can use the RRP check command and the public can use WHOIS. Now, there were three issues related to e-mail specifically that the IAB brought up in its response that the technical review -- these were delivery to nonexistent dot com or net domains that now required additional processing to contact the SMTP bounce server. Also, an issue that the IAB related with misconfigured MX records causing a situation that had previously been a soft failure that would not have caused a message to bounce in the presence of the SMTP bounce server turning into a hard failure. And then finally mail clients or mail user agents with misconfigured SMTP servers that attempt to submit mail to the bounce server got a misleading -- potentially a misleading error message. So, we took all this feedback very seriously and thought very hard internally about ways that we could -- ways that we could work around this and also the community presented some valuable suggestions and one of the things that we are considering is a change that we think addresses these issues in really I think a pretty good way. That is the addition of a wildcard MX record with a nonexistent target domain name to the dot com and dot net zones. For example, as you see there. And just to illustrate my previous point, the first time I put this in the slide I had the domain name on the right here as doesnotexist.com then I thought, I better check make sure that that is not in fact registered before I put it up on the slide in front of a room full of people. And sure enough someone had registered doesnotexist.com I had to find a different name. It would be a wildcard MX record whose target mail exchange entry simply does not resolve. Along with this we would stop the SMTP bounce server, would not run it any more and connections to TCP port 25 would be reset. So, looking at these issues in the context of that potential solution, I'd like to go over what our initial analysis shows and I guess make a request for additional feedback on this from the community. In the case of delivery to nonexistent dot com or dot net domain names a server that is compliant with server 2821-compliant server, which is latest and greatest RFC is going to query for MX records, going to receive the synthesized response resulting from the wildcard record, and according to RFC 2821 it's supposed to report an error when all the MX records are unusable which they would be, because the target would be nonexistent.

So, in our testing we found that recent versions of the most popular mail transport agents the list here, Sendmail, Exim, Courier, qmail and no particular order, they all treat that as hard failure hand bounce the message back to the sender. That would be the desired behavior. Postfix does treat this as soft failure and requeues the message. The advantage of the record in this situation is it does move the processing back to the SMTP client and it eliminates any dependencies on the SMTP bounce server. So, therefore, any -- the performance of the bounce serve we're have no bearing on the solution because it's not required any more and the solution, others had mentioned possible privacy concerns about the bounce server and although we said very clearly that we don't even have log files there to collect any information and our privacy policy specifically mentions this, the situation completely goes away because the bounce server would not exist any more. The next issue was misconfigured MX records. Just to restate this issue was the situation where a domain would have multiple MX records, one of which would have a typo that would cause it to be a nonexistent domain, however in the presence of the wildcard that MX record would not be rejected by a mail server, a sending mail server now the MX record rather than having a nonexistent target now becomes usable and the mail server would attempt delivery to the bounce server. And the bounce server then would respond with a 5xx error indicating that the message couldn't be delivered and as I mentioned a couple of slides ago, this turn what had previously been a transient failure into a permanent failure cause the message to bounce. So, after this issue was reported we took a look at -- looked at all the dot com and MX records. We did a query for every single registered entry at dot com and net. The thing that we took away this section is extremely rare condition compared to some other situations.

So just to give you some data we thought was interesting and worth sharing, the number of MX records that lead to known unroutable addresses that is when you resolve the target of the MX record and look at the address the numbers that are -- the number that have known unroutable addresses is just over 6%. The number of MX records that have IP addresses rather than domain names as target a common error by people who don't understand the proper syntax of MX records that came in at 1.5% of total MX records. And then the number of MX records whose targets were nonexistent domain names in dot com or dot net was 0.7% that is roughly 2,000 times less than 1.5% number there. But in any case, with the elimination of the SMTP bounce server, we believe this would no longer be an issue because now misconfigured MX records become unusable. Although in the presence of the wildcard record, the target of the MX records would still resolve, there would be no bounce server there to accept any connection, the connection would be refused. So, we have only started to do some testing on this. But it is our initial presumption that mail transport agents react more favorably in this situation, namely a reset connection than they would to some other 5xx response. Let me say that one of the other solutions that we heard from the community was why don't you keep the bounce server but just run in such a way that it returns SMTP 554 which is the way within the SMTP protocol to formally reject projections. We looked at that and also looked at the wildcard MX solution. Based on our preliminary analysis the solution I'm describing to you here, namely wildcard MX record with no SMTP bounce server running at least based on our initial analysis seems to be the best solution. Then the final e-mail related issue identified was that just to recap that, a mail client would a misconfigured outgoing SMTP server that previously would have been nonexistent in the presence of the wildcard A record would resolve that mail client would have submitted mail to the bounce server would have gotten potentially misleading error message saying that the domain doesn't exist. With this solution that I've been describing, the bounce server isn't running any more so that mail client would get a connection refused when it tried to submit mail to its misconfigured out going SMTP server. And again it's our presumption that based on initial testing anyway that this situation is more favorable than a bounce server that would return 554. So, that's my presentation and I will turn it over to Ben.

Ben Turner: what I'm going to go through is how consumers reacted to the service post-launch. So what we did is we went out and we fielded three different sets of surveys, or research. we did one in the U.S. where we selected a random sample of 1,027 consumers.

That sample set had a plus or minus 3% margin. The vendor we used to do the research is a company called Markitecture. It does a research for a lot of the leading consumer package companies for different technology companies. The second society of research surveys we did was yet another again segment of the U.S., 1,000 Internet users randomly selected that had margin of error of plus or minus 5%. The company we used to do that was Customer Strategies. And then the final set we did were international we did one in the U.K., one in Germany and one in China, each one of those -- the users with the plus or minus --

John Crain: can the person if you can hear me put us on hold? Take it off, please? You probably can.

Ben Turner: so, the 300 end users in each of those markets were surveyed that gives you a sampling error of plus or minus 3%.

So, what did we find out from this research?

Basically what we found out is that there was strong preference for the SiteFinder error page versus the current error page. The U.S. 84% had preference in the U.K., it was 81%. in Germany 61%. In china it was 76%.

The other interesting column to look here or row to look at here is the last one which is strongly preferred the current error page, that you can see the U.S. 5%, 6% in the U.K., 15 in Germany and 10 in China. So as we went deeper in understanding the research, the other thing that we found is that 76% of Internet users rated the page excellent or very good. So the way to look at this chart is, is the column on the left gives the total view of the site or the SiteFinder page where you see the 76%. Then each of the rows or columns to the right gives you each of the areas. The search box 34% that use search box rated that.

Excellent or very good. Web suggestions or the digi-spell correction 38% and popular 45%. The two key numbers here, the take-away numbers are the summary numbers which is 76% rated the site excellent or very good.

And 4% rated it poor. The next finding out of this was that over 60% found the site easy, convenient and useful. And again this breaks into the different areas, the different areas on the site. On the far right the two columns show the difference between light and heavy Internet users. so on the light side 59% found it was easy, convenient, useful.

While 67% of the heavy Internet users found it easy, convenient and useful. We found to be an interesting differentiation. So the other thing we found is that over 50% said it makes the Internet better, while 3% said it did not at all. So if you take a look again at this chart it's laid out exactly the same way the far left column gives you the summary. The 53% shows you the extremely and very well. Bottom row does not describe at all they thought it did not improve the Internet at all. I'm not going to read to you completely but this gives you a sense of some of the direct feedback or the testimonials we got from doing the survey. The top one comes from a guy at Mercy Behavioral Health.

And then basically what that says is that there's more functionality than you get with a 404 so it's helpful for me. And what I'd like to do now is turn it over to Rusty who will wrap things up.

Rusty Lewis: thank you, Ben. And good afternoon everyone. And thank you to Steve Crocker and the SECSAC and ICANN for giving us an opportunity to present our data regarding the SiteFinder service.

We're happy to be here today because it's an important conversation we're having. As is the case with any new service the provider has to listen to the marketplace to be successful. We spend a lot of time today talking very openly about the issues we've encountered since launching this service. We've had a lot of constructive input from many people in the technical community. And we're very grateful for that. Believe me, we want to assure you that we are listening, we do care, your input is important to us and we plan to act upon it. What I'd like to do is just share some of the next steps that we're considering at the present time.

If and when we relaunch the service, we've gotten several specific action items that we're considering and we'd like to solicit further input. Many of these issues, Matt and Scott have covered in their presentations.

First, response to a number of requests that we've received for advanced notice and that was also reinforced I think in Steve's opening comments as well as in the technical review board comments that Scott Hollenbeck covered. We've believe that it is appropriate to give the community advance notice at least 30 to 60 days before we would re-launch the SiteFinder service. Now, I think it's worth noting that many thought before we launched the service that the presence of wildcard in the DNS would somehow break the Internet. Well clearly the evidence does not support that at all quite to the contrary, properly configured applications do work, our experience when the service was live that we were able to continue to process over ten billion queries each and every day with 100% availability in the com/net zones. The number one complaint actually that we heard across the board was this lack of notice. And we do sincerely regret any inconvenience that the service may have created.

And notwithstanding the fact that most of the concerns that we've discussed today and the ones that we're aware of were largely addressed in a matter of days if not hours, we do believe that a longer notice period would be appropriate. Second, as Matt Larson went into some detail we do believe the addition of a wildcard MX record would address many of the misconfiguration issues that crop up, in e-mail applications, as well as some of the privacy concerns much we did make it clear that we don't retain or collect any data, but the existence of a wildcard MX record would put that to bed once and for all for the privacy community. Third, we talked about the needs of non-English users.

And we do feel that localizing the version of SiteFinder for the international community is certainly an enhancement to the service that's well worth doing and it is one as I believe Scott or Anthony had mentioned in their presentation that we do have plans on implementing localized versions of the SiteFinder service. And finally, based on all of the input that we've received since we've launched the service, we do want to go back and update our white paper and we'll be looking for feedback from the broader community over the next few weeks. So, the bottom line is we're very open to suggestions from the community, we are going to continue a broad out reach effort to get additional comments and we welcome your thoughts. I also think that it's important that we sort out how standard compliance services like SiteFinder can be launched. From our perspective, we struggle with the notion of what's the point of standards, what's the point of best practices, if those who choose to ignore standards are favored over those who follow the rules.

Clearly, we know other registries are interested in exploring similar services, and we believe it's important to develop best practices to promote and encourage uniformity and consistency. A stable, a secure, an interoperable Internet we believe is, in large part, dependent on standards and best practices. Finally, we believe that encouraging innovation at the core is just as important as encouraging innovation at the edge. If that is not encouraged, it's going to result in less investment, less R&D dollars that go into the Internet infrastructure and we believe, ultimately, a weaker Internet. We believe that's a problem that should concern not only registry operators but policymakers, the technical community, and all of us that are worried about the growth and prosperity of the Internet. So I'd like to again thank the SECSAC and ICANN for the opportunity to present our supporting data on this service. We look forward, Steve, to reviewing any additional information that your committee may receive. And also we're looking for additional feedback from the community to many of the thoughts and suggestions that Matt Larson has raised in his presentation. So with that, let me turn it back to Steve for questions and answers, and I'd ask Ben Turner to come up who will serve as our MC for the Q and A section of this presentation?

Steve Crocker: thank you, Rusty.

All right. Let's see. It's now 2:20, and the agenda -- the schedule that I put up before is no longer operative, as we used to say. Let me ask you guys who are here, do you want to take a break now or do you want to roll right into questions and take a break later? Quick show of hands?

>>>: break.

Steve Crocker: yes, raise your hand if you want to take a break now.

Too bad.

(laughter).

Steve Crocker: okay. I think, then, we are primed here and ready to go. As I had said, wasn't to ask people on the phone first, come back to the committee, then back to the phones and then we'll take questions from the people who are sitting here patiently.

All right. Anybody on the phone?

Ram Mohan: Steve, this is Ram. I have a question.

Steve Crocker: go ahead.

Ram Mohan: the question is why did the SMTP problem not show itself during your initial tests, since it seemed to be a basic error that would have shown itself early and often?

Scott Hollenbeck: Ram, Scott Hollenbeck here.

One of the things we found were that a large number of the error situations were typically due to misconfigurations and as Matt said in his presentation, the actual incidence of the misconfigurations is actually fairly rare. So while we did anticipate some misconfigurations, a lot of them were due to situations that we didn't become aware of until they were brought to our attention post launch.

Steve Crocker: any follow-up? I'll ask a question just to follow-up on that.

One of the things that becomes clear from the sequence of the presentations is that you designed a series of experiments, test, research internally and went out and executed that beforehand in contrast to the more public kinds of processes where things are designed and tested externally, shared among the community. Do you have any numbers on how much your testing did compared to the quality of the testing that takes place by those public processes? That is, you have 95% confidence levels for, you know, so many thousands of responses. Do you have comparable numbers for what the depth and breadth of testing that takes place by the usual public processes?

Ben Turner: I'm not sure there are usual public processes for testing. What we did is we followed, you know, one version of best practices to test, and we're fairly confident in the way we did that testing. It's no different than the way Verisign tests the majority of the products that we roll out.

Steve Crocker: anybody else on the phone want to follow up on that or anything else?

Ram Mohan: Steve, this is Ram again. I have a different question.

Steve Crocker: go ahead.

Ram Mohan: other than the user interface surveys and tests, what other non-web-based tests were conducted? That's one question. And the second is what kind of testing was done for user applications as compared to user interface.

Matt Larson: this is Matt Larson, I can address that as a stall for time moving quickly back through the presentation. One of the things Anthony touched on was this, and I'm looking for it.

Here we go. We had a third party, and the slides aren't numbered it's from Anthony Renzette's presentation, the title is SiteFinder Development Research, Third-party Testing. And we engaged a third party to do some testing, and the numbers are on the slide here. We had them test -- identify different application protocols, the most popular ones.

Actually, they identified the list. We worked with them to do that, and then they tested, it was 53 implementations of these 37 different protocols that fell into 13 different categories. And they did that testing to evaluate the impact that a wildcard would have as well as these were all network clients, so what the behavior would be when those clients connected to the wildcard address and got either a TCP connection reset or a UDP port unreachable. And their findings was that the user experience would not change dramatically.

Steve Crocker: any other questions from the phone?

Let me move to the panel, and then -- anybody on the committee want to ask questions here?

David Conrad: this is David Conrad. A quick question about the survey itself.

Steve Crocker: speak up a bit, David.

David Conrad: I'm sorry? Are there any plans to actually survey against -- instead of the error page versus SiteFinder, the SiteFinder versus AOL Search or MSN Search?

Ben Turner: so when we did the survey, we caught all the varieties of error pages that exist, so of the existing error pages, what was your view of SiteFinder versus the existing. That would cover MSN, AOL, the whole variety.

Steve Crocker: let me follow-up on that. Do you plan to make this data available? I mean, you presented summary information here, but the experimental design and the instrument you used to do the survey and the responses and so forth?

Ben Turner: the end-user surveys that we did, the results we just took you through?

Steve Crocker: yes.

Ben Turner: what we can share with you is what we walked through. We can give you the additional tables, if that's helpful.

The actual, you know, feedback that we got directly from doing the surveys, you know, is proprietary information to the company, and just like most companies, we're not comfortable sharing that in an open forum where we have competitors involved in the process.

Steven Bellovin: what the questions themselves? The wording of the questions are very critical in the answers.

Ben Turner: I think what we have believe we've done is a very methodical set of surveys. We used third-party researchers to do that that are well qualified, and I'm not quite sure what concern you're getting at there.

Steve Crocker: let me jump in.

Within a stone's throw of where we're sitting is heart research well-known pollster. I hung out there just to see how things are done.

It's eye opening. Polling is a proactive, not a passive operation, and the structure of the survey is all important. And many times, and I have no idea whether it applies here, but many times, polling instruments are put together for the purpose of conveying a message and what's called push-pulling as opposed to getting a neutral answer.

There's a very, very big range of results you get depending on whether you're putting together a carefully engineered, carefully designed experiment in an academic setting versus a market-oriented mechanism.

There's no way to tell from what you've presented here, unless that gets opened up and examined.

Ben Turner: there actually is a way to tell, and let me show you.

So the way to tell is you see multiple responses within a question. You see multiple samples. You see multiple responses within a question, you see multiple questions. What you're talking about, I could go out and design a one- or two-question survey and get an answer, but the fact that we designed a survey or market research company designed a survey that had multiple questions with multiple answers for each question eliminates the concern you're talking about. In addition, by having a truly random sample, you're not going to be able to stack the deck, if that's what you want to call it.

Steven Bellovin: it's not even a matter of stacking the deck. It's a matter of what is it you're measuring? That's why I want to see what the questions are. It's a very subtle business.

Rusty Lewis: this is Rusty Lewis. if I might just address this issue.

As I mentioned in my closing remarks, the success of any service really depends on what the reviews are from the market. And from our perspective, we did not conduct this survey for purposes of convincing the SECSAC or ICANN or anybody else, for that matter, that end users want to use this service. We're a commercial company. We are interested in getting unbiased, unfettered feedback from the marketplace so that we can design services that meet commercial needs. I mean, so clearly we're presenting here data after the fact that we did before the fact to convince ourselves that it made sense to roll this kind of service out, and that, in fact, it would be useful to the end users. So I can assure you that from the standpoint of the way in which we went about hiring the best people we possibly could and to get the kind of information, Steve, that you're getting at, we want real feedback. Just like in this whole process, I mentioned the fact that we've gotten a lot of constructive feedback from the technical community on ways to improve the service. We are sincerely grateful for that. We want to make the service work for the end users, and that's what we think is the value of this community input.

Steve Crocker: and so, it's fine. Did you ask the community of prospective -- of users who are going to be affected in what ways will this break your existing applications or interfere with your existing work? Sort of the complementary question to the ones of "do you think you would like to see a search service?"

Ben Turner: I think if you look, to give you a sense, we would not ask the question if we did not want to have a fair, balanced, you know, understanding of what users really thought. "Do you think this is poor?" this clearly shows we're asking those types of questions.

Steve Crocker: okay. We should move on to other things.

Anybody else on the committee? Ray.

Ray Plzak: I want to go back to this right here. You're asking people about the web page itself, the usability of the service once they have arrived. And I could do a similar thing to other search services and probably get a very similar set of responses.

And so I'm kind of curious about the usability of this web page, how is it germane to the other things that we've been talking about here in terms of DNS and SMTP?

Ben Turner: I think Verisign's view is the Internet is made up of multiple user sets. And one of those user sets are consumers who actually use the Internet to surf, to buy things, to send e-mails.

That's what we were testing with this survey.

If you guys don't think consumers are relevant, we can move past this data set.

Ray Plzak: I didn't say I thought consumers were irrelevant but I'm saying I could go to a similar service, like Yahoo! or whoever you want to name and conduct a similar survey and get similar responses.

Ben Turner: I honestly don't know if you did this survey on another web site what kind of answers you would get. You might or might not get the same answers. I have no idea.

What we tested here was did you find this site useful and did you prefer it over your existing error pages.

Ray Plzak: and what does this have to do with the effect on DNS and SMTP?

Ben Turner: well, like I said, this is targeted towards a specific set of segment that's out there that's using the Internet.

This is the end user, the browser user.

We presented other information that covered the other protocols.

Ram Mohan: this is Ram. Would it be appropriate to say that the survey was fundamentally focused on web users of the Internet?

Ben Turner: the survey we're talking about, that is correct.

Chuck Gomes: and Ram and Ray in particular, I want to respond in particular to your question. My name is Chuck Gomes with Verisign.

One of the segments of the community that has not been looked at in this whole issue, in my opinion, is the user community. I believe they're very relevant. Now, they're not the only user, but they are very relevant. So like Ben said, if we don't care about that part of the community, we should move on. But I think everybody does.

Ben Turner: the other thing, Ram, is we did do a separate set of research where we contacted over 600 enterprises that would not be web browser users; right? So there are clearly multiple segments here that have to be addressed.

Steve Crocker: other questions?

David Conrad: I just wanted to try again because my question wasn't answered.

Specifically, is there any plan for Verisign to do a survey that compares apples to apples, that is the user's experience with SiteFinder versus the user's experience with AOL or MSN search that result from a non-existent domain lookup attempt?

Ben Turner: sorry if I wasn't clear. We did do that research. We went out and asked the question, you know, vis-a-vis the current error page you receive, and we found four major -- five major types of error pages that people saw, you know, what's your views versus those versus SiteFinder. And that's what this data shows, where 84% of the U.S. users, 81% in the U.K., 61 percent in Germany, 76% in China either strongly prefer or somewhat prefer SiteFinder over the current error messages they were receiving.

David Conrad: of those results, do you have a breakdown of the ones who are actually comparing against AOL or MSN or whatever searches versus an error page?

Ben Turner: no, all we did was qualify to make sure they had seen one of those error pages in addition to a SiteFinder page.

Steve Crocker: but you are saying that this data is only for people who have seen somebody else's -- that's not what the title of the slide is. It says SiteFinder page versus an error page versus a SiteFinder page versus a comparable service, AOL or --

Ben Turner: we're liberally using the same error page, the error page covered the 404 error page, Yahoo! error page, Google error page, MSN, AOL.

Steve Crocker: that's the question. You included the error page as to what that would look like as opposed to people seeing only the SiteFinder error page versus the comparable results from AOL or --

Ben Turner: the results would not change very much. Most people are seeing an alternative search error page.

Steve Crocker: so you're asserting that most people don't see 404 today?

Ben Turner: what we're saying is the way we did this survey is we tried to narrow it down so the majority of them saw a comparable search page. Error page is probably what's causing the confusion.

Steve Crocker: right.

Ben Turner: we're using that in a market research term, not in a technical sense.

Steve Crocker: right.

David Conrad: however, in the testimonial and verbatim, the very first ones said something about a 404 error.

Ben Turner: right. So we didn't completely eliminate the 404 errors.

The ones we did of a thousand users, customer surveys -- customer surveys to do, we asked them specifically to give customer testimonials. The data you're looking at that shows the preference here is based off the Markitecture research where we narrowed out the 404 error page.

Steve Crocker: thank you. Let me move to a general open forum and allow people who are here to come to the microphone and ask questions.

And it's also been suggested that I announce once again the e-mail address for comments to come in which we're monitoring in real time.

secsac-comment@icann.org. That's secsac-comment@icann.org. Thanks.

Marilyn.

Marilyn Cade: I have two questions. I'm looking at the objective which says this is to address the needs of consumers and SOHO's in Europe and the U.S. You undertook this research as a registry; is that right?

Ben Turner: there's a separate set of research that was done before we launched the service. So the research we just talked about was post launch.

We also, just like we do with any product in Verisign, do pre-launch research, and we did go out, and I think you're talking about -- we did go out and do, as a standard course of business, you know, the naming and directory services decision goes out and does surveys with businesses and end users, consumers, to understand what it is they're looking for with respect to their usage of the Internet.

Marilyn Cade: and you undertook this research as a registry; right?

Ben Turner: no, we undertook this research as the naming and directory services division of Verisign.

Marilyn Cade: okay.

Verisign is a registry.

Ben Turner: actually, no. To be precise, it's a business unit within the division. The division is naming and directory services division, and that's what did the research.

Marilyn Cade: okay. so in any case, I think we're sort of getting close to my ability to narrow down an answer to this.

So you were studying the behavior of end users on the Internet. And I have one more -- consumers and SOHO's, and what they were looking for.

Ben Turner: right.

Marilyn Cade: and that was so that you could develop a service yourself or so that you could provide information to registrars whose job it is to actually serve registration services to end users.

Ben Turner: we actually did it for both. So what we did is we did this research regularly to find services we can offer. We also do research to share information with registrars on what we think will help them sell more product.

Marilyn Cade: okay. I didn't -- and perhaps I just missed it. Has this data already been provided broadly to all of the competitive registrars?

Ben Turner: actually, what we do is when we do research that's relevant to the registrars, we provide it to them.

Marilyn Cade: to all of them.

Ben Turner: to the ones that want it. It's available on our web site, actually.

Marilyn Cade: so all of the competitive registrars would have access to all of the relevant information, which is about the development of services that touch end users; right?

Ben Turner: no, not all of it, because some of it has nothing to do with registrars. The information we provide registrars are things that are relevant to the registration of a domain name.

Verisign runs multiple businesses. We do research to understand what's going on in the market. We take the appropriate research and apply it to the appropriate businesses.

Marilyn Cade: but if I were to ask the competitive registrars, they could go to your web site and find some information, with whatever the caveats are that you're describing.

Ben Turner: right. And I think you would find most of them have attended seminars we've held where we bring in the market research company to walk them through the results.

Marilyn Cade: okay. Is this data presently available to the competitive registrars with the caveats that you've provided?

Anthony Renzette: this is actually used in a number of ways and been alluded to in registry meetings, and we've had the market company come in and give them the data directly, most of the time it's given raw.

Most of the registrars who work with us in that capacity have gotten a monthly newsletter from us, which includes information on renewal rates, things that influence and so forth. All of those things come ultimately from research. So if there's any other questions about how we conduct research, we can do that off-line, I guess.

Steve Crocker: a number of things are queued up here. I'm a little lost. I beg your indulgence but I want to go back to Marilyn's question, basically.

I think she was asking the registry versus registrar distinction, and your answer was this is a business unit of Verisign that's going out and doing this work. But from our perspective, the world is divided up into roots, top-level domains, registries, registrars, and then others, in a sense. And as we all know, one of the things that's at the core of the inquiry and the discussion here is the line that has existed very cleanly for a long time between registries and registrars. And so it's helpful to know from your perspective what actions and decisions and how these things are cast, map onto the registry versus the registrars.

Ben Turner: so we're all clear, SiteFinder service is not a registry service. It does not provide a service of registration of domain names for registrars. So, you know -- and there -- I'm a little -- I'm trying to understand where you're going here.

Steve Crocker: so this is a very core piece, absolutely core piece of Internet technology. A registry has two sides to it, basically. This is just sort of registry 101, if you will.

One side of it is putting new entries in or taking entries out of the registry itself. the other side of it is the lookup process, the serving of queries that come to it. That is sort of absolute, that's why it's there. The numbers you served, what was it? 300 billion lookups during some period of time, that is the heartbeat, the high-speed interloop, if you will, of this operation. Architecturally, there is just no question that when you are making a change to what comes out of that lookup or the way that lookup is dealt with, that's a registry operation. At least I don't know another way to look at it.

Ben Turner: actually the way it's looked at, and it's actually very clear, is you should do that in an RFC compliant way. Which is SiteFinder is completely RFC compliant.

Steve Crocker: so I don't want to skirt away from this. One can have a discussion about whether or not the change is within the RFC --

Ben Turner: the question is are we or are we not compliant within the RFC wildcard?

Steve Crocker: I don't want the change the question. Is there a change to the registry and what actions did you take with respect to that as opposed to what actions did you take with respect to registrars or other external services?

Ben Turner: in the definition of registry operations that you're using, the answer is no, there was no change to the registry operations.

Steve Crocker: how do you reconcile inserting a wildcard into the domain and serving up an alternative response to a query, to a lookup, and say that's not a change to the registry? I mean, that strikes me as not a debatable issue or one that's subject -- it's just a straightforward fact.

Ben Turner: what we have done is we have done an RFC compliant wildcard implementation, which is what we're required to do is to stay compliant with the RFC's.

Steve Crocker: I think there's a useful discussion about the use of wildcards and how they get treated and I acknowledge that.

Ben Turner: that's what I thought today's meeting was about. if we're going to have a discussion about your definition or ICANN's definitions of registry services, we're basically going to have to stop the conversation and turn this over to a different venue. We're not going to have this discussion now.

Chuck Gomes: can I ask a process question, Steve?

Steve Crocker: sure.

Chuck Gomes: I'm totally confused about how what we've been talking about the last few minutes has to do with security and stability.

Steve Crocker: it's crystal clear. If you make a change to the registry, what is the impact of that, what are the ripple effects?

How does that change the interaction?

Chuck Gomes: we've had no changes to domain registration services.

Steve Crocker: I don't know what domain registration services is intended to convey, but is there change to the registry operations and the answer is unequivocally yes.

The questions that follow from that is does that matter, is that an important change, how was it done, a whole series of other questions, acknowledge all those questions. But there can be no question that was there a change made to the core registry service.

Ben Turner: there was an RFC compliant change made to the registry services which is well within the purview of the registries.

Steve Crocker: so let me translate that. You said yes, we made the change but we believe we made the change within the allowed range that the protocol definition -- the RFC specifies. Is that a fair reading?

Kevin Golden: hi, Steve. My name is Kevin Golden. I'm counsel to Verisign and I think the difficulty we're having here is certain terminology is being put out there that is loaded with legal significance.

And so there's a reluctance to go down that path in what should be a technical discussion.

So we're happy to have a discussion on the legal nuances of some of this terminology with ICANN's counsel, and have done so on many occasions, but I don't know that it's productive to continue going down this path because it's loaded terminology, and I think that's what you're seeing.

Steve Crocker: I see. thank you.

All right. So with that I think it's fair to move on to other kinds of things. Jim, you've been monitoring some things coming in from secsac-comment? And I'm sorry, are there people queued up for the floor? Let me do that first.

David Lesher: good afternoon, David Lesher with NRK Research.

I'm looking at the page labeled issues analysis but unfortunately does not have a number but it was reasonably early, and it says issues more likely to occur with at least moderate impact and how addressed. And I count, one, the first item and the last item as being addressable by Verisign. And I count one, two, three, four, five in the other hand that have to be addressed basically by the rest of the user community.

My question is what's the estimate as to the cost impact and the time to develop these answers and deploy them.

Ben Turner: what we looked at here is how we mitigate.

David Lesher: not the cost of mitigating.

Ben Turner: we did not go into the detail.

David Lesher: next question on the TRP work product, there's another question that says some delays, some things back and forth but in the long run just sort of say not important. Has there been any analysis as to the total additional bandwidth or total additional time this will take on a worldwide basis?

Scott Hollenbeck: from the TRP's perspective, no.

What you see here is the total what have was discussed by the TRP based on their analysis of impact.

David Lesher: is it fair to say in the words of the current administration this is a cost shifting, from Verisign to the community?

Ben Turner: I think that would be unsubstantiated unless you --

David Lesher: I'm addressing the things stated here.

Ben Turner: these are the issues we looked at and what we looked at, Scott went through.

David Lesher: but there was no or very little detail as to the work actually required to solve the problems this creates. It just says there's a minor problem. Well, minor depends on how big your sample set is. It's a big sample z that's my only point. Thank you.

Steve Crocker: question from the committee here, both Steve and Paul are queued up. Let me take Steve first.

Steven Bellovin: I've got several questions. One, of the three anonymous reviewers, were any of them from the IETF leadership, work group chairs and so on other than the Verisign people?

Scott Hollenbeck: no. When we asked people with those sorts of affiliations if they're interested in participating, their answer was no.

Steve Bellovin: okay. it's too bad since that's where the center of expertise in protocol analysis.

Scott Hollenbeck: we made an attempt to reach out to those people and a few of them, because I think of the controversy of this, said we don't want to be involved at that level.

Steve Bellovin: I'm not blaming you. I'm saying it's unfortunate.

Question of software updates, there have been a number of surveys in the last three years or so on how long it takes people to make upgrades even when there are critical security problems. and a remarkably high percentage of people never upgrade their machines. And even for security things, even for people who should know better, such as SSL, SSH, servers all show the same patterns. These are things run by more sophisticated people than ordinary consumers. I heard a number, I think it was 40% to 60% of the users never even clean up their infected home PCs when they're infected. So do you actually expect people are going to make any of these software changes or simply wait to buy a new computer with whatever Microsoft ships in years from now?

Scott Hollenbeck: the discussion the panel had around this would probably answer this no for the reasons you just described. There are lots of people out there running old operating systems for various reasons.

From the panel's perspective, they were trying to discuss what the mitigation strategy would be, not the likelihood of the mitigation strategy being viable. Does that answer your question?

Steve Bellovin: it turns to a previous question about cost shifting but yes, it answers my particular question.

Chuck Gomes: let me add one more thing. Chuck Gomes again. There were a total of four companies that actually contacted us when we made information available to do that, and identified specific problems and we assisted them with those.

If there are companies or individuals, for that matter, that have issues that they can't resolve, we do have an e-mail address that's been made publicly available where they can contact us, and we will help them with it. But what you're hearing, we need -- what you're saying is all good and fine, but we need the evidence of it to be able to work with it.

Steve Crocker: did you write RFC's that lay out the mitigation strategies?

Chuck Gomes: no.

Steve Crocker: thank you.

Did you have more, Steve?

Steve Bellovin: I think that's it for now.

Steve Crocker: okay.

Scott Hollenbeck: Steve, if I could answer your question partially, no, we don't have an RFC out, of course if we did it would take far longer than a few months to get it published. We do have a guidelines document that we wrote and released a few months ago. This is a document that recognizes that registries are doing this. There are issues associated with wildcard, and we are looking at a document, and using the formal processes using the IETF processes.

Steve Crocker: I'm glad you're looking at it. I commend that process to you.

Thanks.

Paul.

Paul Vixie: yeah, first I have a question for Matt and then I have one for Ben Turner.

So Matt, in your slides you said that one of the things you were considering for the future of mail related to the SiteFinder and wildcard was to put in an MX record pointing to a nonexistent name.

Thank you for finding the slide. That particular name would be found because a mailer, which didn't get the glue, would then query for an A record by that name and they would get the SiteFinder name.

Matt Larson: not unless it were delegated to a zone who's a apex had a (inaudible) domain.

We could do it in an undelegated TLD.

Paul Vixie: I'm not suggesting you do it at all. I just wanted to clarify that point, since there is a wildcard -- or would be in this scenario, a wildcard under dot com, you would not be able to find a name that doesn't exist so I thought your slide was inaccurate in that minor technical detail.

Matt Larson: you're right, I did not say this would have to be delegated, and it would be a -- it would be a negative response but not strictly speaking a nonexistent domain response.

Paul Vixie: that would be an empty response; strictly speaking, a nonnegative response. but that's okay.

Ben, I have a question. Actually, I don't know who at Verisign to address this to, but your slides came closest to this point. You surveyed some number of web browsing end users in order to find out what they thought the impact was on their web browsing experience; however, other than by accident, that population is not your customers. And what I'm now curious about, you have two hats. As a registry, your customers would be other registrars. As a registrar, your customers would be other registrants. Did you survey those two populations to find out what the impact was on them; in particular, the registrant population is the one that's affected because they are the ones that are -- whose names are being mistyped.

Ben Turner: yes, I don't have this -- I don't have the specifics in our end-user surveys that we hit registrants. The probability is we did hit registrants.

Paul Vixie: so you didn't specifically survey any of your own customers but, rather, just essentially third-party end users who happened to be using web browsers were the targets of your survey?

Chuck Gomes: I'd like to make something very emphatically clear. Verisign naming and directory services is the home of the Verisign com/net registry with which I'm associated. The Networks Solutions registrar is not associated with our division at all.

Paul Vixie: Chuck, thank you for that but I want to ask a follow-up question. It's still a wholly-owned subsidiary?

Chuck Gomes: that is correct.

Ben Turner: what I'd like to understand, though, is what does this have to do with security and stability and what does this have to do with wildcards?

Paul Vixie: I just wanted to know if you had tried to survey any of your own customers, and it sounds like other than by accident, you didn't. That's all I wanted to know. Thanks.

Ray Plzak: as a follow-up, I would like to ask you the same question about your survey of the web users.

Ben Turner: and that question would be what?

Ray Plzak: the one you just posed.

Ben Turner: I'm confused.

Steve Crocker: it's probably helpful to restate it.

Ray Plzak: his question is what did the things that Paul have to do about security and stability in wildcards?

Ben Turner: my question was with respect to there's a subsidiary of Verisign called Network Solutions and there's a division within Verisign called Naming and Directory Services which does not encompass Network Solutions. What does that have to do with this discussion?

Ray Plzak: thank you for the clarification. I'll ask something else later.

Paul Vixie: Steve, I'll follow-up on that. I wanted to get clarification on the survey that had been done. Just wanted to make sure I understand what had been presented.

Steve Crocker: so go ahead.

Paul Vixie: I have what I need.

Steve Crocker: thank you.

Let me poll people on the phone. is there anybody who has been listening who wants to ask a question?

Ram Mohan: Steve, this is Ram. I have a couple of clarifications regarding the survey.

Steve Crocker: go ahead.

Ram Mohan: the first question is, the survey shows a comparison of the SiteFinder, the web page. The first question is was the SiteFinder comparison made in comparison to MSN search, which is a default in Internet explorer, or was it in comparison to a 404 error page, the first question.

The second question is did the two survey vendors that were used, did they use the same methodology? I thought I heard that the same methodology was used and later I thought I heard maybe a different methodology was used. A follow-up to that, are the results that are put out a blended result of the two surveys or the different results from the two surveys?

Ben Turner: your first question was already answered.

Your second question was did we include -- was it -- I'm confused on the second one. Was it blended data? Is that what you're asking?

Ram Mohan: yeah. You used two vendors, and I thought you had said earlier that the same methodology was used, and the results -- I'm unsure or unclear as to whether the results that are presented, whether that's across the various surveys done by different vendors or is it individual, independent results?

Ben Turner: it would be invalid if we blended the results so we did not blend the results.

So we used two different research companies. One did a U.S. sample.

That same one did a U.K., Germany and China sample. That's the information that's displayed in the charts. the other, customer strategies, did another U.S. sample. We will asked them to gather testimonials, and that's on the testimonial page.

Steve Crocker: let me ask a question within that, a question from Thomas Roessler. What portion of those surveyed in Germany and China did not speak English?

Ben Turner: I don't know the answer to that off the top of my head.

Steve Crocker: all right.

More questions from the floor.

Rick Wesson: Steve, this is Rick Wesson and I have a question.

Steve Crocker: Rick?

Rick Wesson: yes. and this is, again, for the survey. My question is why does a survey of users show impact on security or instability versus a survey of applications? If Verisign could just discuss why they chose to survey users versus applications.

Steve Crocker: the question is on the screen there.

I think we asked that a little bit earlier as to whether the survey had focused on security and stability issues. But you may want to add to that.

Rick Wesson: the question is specifically why they chose to survey users versus applications and why stability or security affects users and not applications. I take it there's some philosophy that Verisign has on this.

Ben Turner: we actually did survey application users in our launch testing, and then we also made available other ways to get information to us regarding impacts on applications post launch.

Rick Wesson: if you surveyed the effect on applications, where is the raw data? Or can you make the raw data available?

Ben Turner: yes. So the current slide summarizes the application testing we did. So if you look up here, you can see that we went across 13 categories, 37 protocols, and 53 different implementations of protocols.

Steve Crocker: every time I see that, I'm inclined to ask, did you engage the open community, the IETF and operators and IAB and so forth. And I think the answer, as fast as I'm inclined to ask that, I guess your answer is you view this as a proprietary process.

Ben Turner: in that we did, you know, open it up to -- we went out to 600 companies, asked them if they wanted to participate. We had 55 that were interested in participating. Out of that 55, 35 actually wanted to test.

Steve Crocker: I'm struggling here, but the thought that is -- that wants to get out is that the Internet was really built not by a set of companies that built a closed, proprietary system but by a much more open process than that with, you know, arbitrary numbers of individuals coming from every quarter and dealing themselves in as they want rather than being selected.

Rick Wesson: Steve, would you please ask the Verisign people to make available that protocol survey?

Steve Crocker: I think you just did.

Rick Wesson: thank you, sir.

Steve Crocker: so that's the question. Would you make a protocol survey available?

Ben Turner: sure. What we've made available here are the results. The survey itself is proprietary and just like other companies, we're not comfortable making that available in an open forum.

Rick Wesson: just to clarify, so the data would not be available for outside evaluation; is that correct? The raw data.

Ben Turner: I think throughout this process, we've been willing to sit down and discuss the data we've collected. That's what we're trying to do here. If you have specific questions, it would be helpful if we could know them and we could try to answer them for you.

Rick Wesson: the specific question is can I have the raw data? Or can the raw data be made public?

Ben Turner: I think I just answered that question, actually. Really, if you have any specific question --

Rick Wesson: (inaudible) I can't pick it up on the web thing, so if you could just comment; I'm sorry, I didn't catch that.

Steve Crocker: Rick, I realize California is different from Washington, so let me use the skills learned in Washington to translate what you've heard here, and the answer is no.

(laughter).

Rick Wesson: thank you.

Steve Crocker: Olafur.

Olafur Gudmundsson: hi, I'm Olafur Gudmundsson. First question to Ben or comment, you're absolutely correct, adding wildcard to (inaudible) is fully RFC compliant. No question about it.

The issue is for 13, 15 years, the com had been without wildcards, and then one night it appears without a warning. Is there -- I hate to use this word I'm going to use, because I think your lawyer is going to come back and hit me in the head. Does this non-use of wildcards for 13 years mean that there was an unpublished service-level agreement with application writers and users of the com zone that this would not change overnight?

Ben Turner: no, we look to standards as the guidance.

Olafur Gudmundsson: standards are not the full answer to everything.

There is social behavior also involved, there is courtesy, there is a kindness to others.

I've been trying many years to have wildcards thrown out of the DNS standards. Every time people tell me not to do it but every time people start using them there are more problems that come up. So yeah, I think the biggest mistake you guys made, and it's really biting you, is the total lack of communications up front. Because the rest of the top-level domains are irrelevant compared to com and net.

Ben Turner: we agree that one of the lessons learned out of this is it's important to go out and do pre-notification on a service like this.

Olafur Gudmundsson: yeah.

I'm going to repeat a comment that I made to somebody at Verisign when they asked me about this a few months beforehand. I worked prior at one of your wanna be competitors, it took me about 20 minutes to kill the project, because a spelling correction is totally hopeless because you don't know what language I'm going to speak. So (speaking in foreign language).

Mark Kosters: Olafur, to be fair, that particular conversation didn't hold up because if I recall, your previous employer did put up a test after --

Olafur Gudmundsson: after they laid me off.

Steve Crocker: you'll help the transcribers transcribe from Icelandic into --

Olafur Gudmundsson: yes, I said I don't understand error messages in English. I said that in Icelandic.

Steve Crocker: they are fabulous but I suspect that tested even their skills.

Olafur Gudmundsson: and Scott and Matt, Internet draft deadlines is either today --

Scott Hollenbeck: Monday.

Olafur Gudmundsson: monday? So get it out so people can look at it, without the note that this is Verisign proprietary information like the current document has, which causes some people not to look at it.

There is already one Internet draft come out that where people are proposing to change the protocol to describe synthesized answers.

Scott Hollenbeck: the guidelines document has no such proprietary markings on it. An earlier version of the implementation document had those on it. We have no intentions of taking that out into a public forum and submitting that to the IETF process.

Olafur Gudmundsson: please do, because it's a good point, and registries who are not willing to do their homework like you guys did, we don't want them to make mistakes.

Steve Crocker: to your point about deadlines for drafts, this is a submission that happens just before IETF meetings and then the window opens up for unfettered input until a similar window several months later.

Olafur Gudmundsson: yes.

Steve Crocker: so I don't know how soon it opens up after an IETF meeting.

Olafur Gudmundsson: middle of November.

Steve Crocker: so that's right after the IETF meeting, basically.

Olafur Gudmundsson: yes.

Steve Crocker: that's just a clarification.

David Conrad: this is David Conrad. I want to clarify at least my view of some of this in that standards compliance is somewhat of a red herring in this discussion. Wildcards are a part of the standard, but it is perfectly standards compliant to create a server that doesn't do anything; right? Saying something is standards compliant is sort of a cop-out from actually trying to interoperate with the technologies that are available on the Internet.

The working group that Olafur is involved in, the DNS XT Workgroup, is not the right place where discussions about the operations and the characteristics of the wildcard implementation within TLD registries.

The appropriate workgroup for that would be DNS ops.

Steve Crocker: who chairs that group?

David Conrad: I don't remember.

Ray Plzak: it's Rob Austin.

Olafur Gudmundsson: I talked to Rob before I came here.

Steve Crocker: does that mean you're speaking for him or he's speaking through you?

Olafur Gudmundsson: he agreed to having this document in an open forum as a discussion point would be a very useful thing.

Steve Crocker: good.

Sir.

Tim Ruiz: Tim Ruiz from Go Daddy Software.

Just a discussion earlier on this, that's regard to SiteFinder's home as an entity, that's with the Naming and Directory services, not necessarily or not the com/net registry business unit. And the com/net registry business unit's only role with SiteFinder is it made the wildcard entries that made it possible.

Ben Turner: that's correct, it's not part of the com/net industry.

What the com/net registry does is the registration of a domain name and the resolution of a domain name.

Steve Crocker: Steve.

Steve Bellovin: I just want to turn back to the question of software -- I'll sit back a little bit. I want to turn back to the question of software updates and timing and so on.

A number of the recommendations spoke about protocol changes to indicate that it's a synthesized response. That at a minimum it was going to take several months through the IETF process. How long do you estimate it will take developers of relevant pieces of software to develop their changes, test them, to test new versions of the software with lord knows what other changes in their environment? In other words, how long do you think it's going to take before these changes can be safely deployed by lots of people?

Scott Hollenbeck: Steve, you probably know better than I do what it would take for IETF standards to be deployed. Typically, that time frame can be measured in months or years.

Steve Bellovin: it will take months at a minimum to get the standard through the IETF for the protocol change. But I was actually referring to the software engineering effort to get it developed and tested, and then the deployment engineering. You guys run very reliable production servers, and I know from my experience you don't do that by upgrading to the latest version of every patch that comes out from your operating system vendor.

Scott Hollenbeck: quite right.

Steve Bellovin: lots of other people have similar concerns, especially about things like mailers, for example. So I'm asking what your estimate is how long a site should test a new version of a mailer before they're willing to cut over to it, one that would have this protocol change incorporated.

Scott Hollenbeck: our standards to deploy a service like that would, typically, as you said, first off initial versions don't get deployed. You let somebody else plow the ground and get the bloody nose. And after that there's still a testing effort measured in weeks or months before we're ready to deploy any kind of new software.

I assume other companies who operate critical services would have time scales measured in the same way.

Steve Bellovin: I agree. I know what -- I know how long we beat up network elements before we put them into our network, and it's an interactive process with the vendor after getting things fixed up in our test lab.

And I'm trying to suggest that trying to get a handle on numbers like this are important if it is decided to go reinstate the SiteFinder service, the timing of the notice that's necessary has got to factor in the delays to let people upgrade and test software, and that's even after everything is defined, which has not happened yet. So I'm asking -- as you say, you test for weeks or months and other people who use e-mail that's mission critical would as well and that's one of the things that would need changes per your own survey.

Ray Plzak: Steve, just a process clarification, my understanding is the way a name gets into the com or net zone, a registrant applies to a registrar and then it goes and gets through process and gets put into the zone; is that correct?

Scott Hollenbeck: not necessarily.

And I say not necessarily because, I mean, at the high level you have the process correct. we receive automated requests from registrars and then names are in our database before they're published to zones.

And there are a number of factors that might prevent a name from being pushed to the zone, such as for example if there is no delegation information, no name servers, it doesn't go in the zone. If the registrar or the registry has the name on an internal status we call hold, it doesn't go into the zone.

At the high level, though, you're absolutely right. The registry gives registrar information, registry publishes information in the zone file.

Ray Plzak: and this originally starts with a registrant of some kind.

Scott Hollenbeck: typically. Registrars register names for themselves where they play the role of registrant, yes.

Ray Plzak: what was the process followed to put the wildcard names into the zones?

Ben Turner: there are no wildcard domain names in the zone.

There's a wildcard entry in the zone.

Ray Plzak: what was the process to do that?

Who made the decision?

And what would have been done to notify people this had been done?

Mark Kosters: I'm having a hard time seeing -- who asked who to pull the plug or insert the plug. I don't see how that's relevant.

Ray Plzak: what's relevant is you're changing domain. Is the domain administrator responsible for it?

Mark Kosters: all the names are registered, there's no change whatsoever, right?

Rick Wesson: mark, I have to disagree with you, this is actually very relevant and interesting question. This change affected vast majority of the Internet. And how many applications consider availability and so it is interesting to ask this question as to what entity put this record into the zone?

I think that's what Ray was asking there.

Steve Crocker: so, yeah, let me try to make this very -- as simple and if it's not if you guys don't agree then we come back again. I think there are two things.

The entity that made changes was the entity that was operating the registry. And as opposed to any registrar and I think that is pretty straight forward. The next question is, does that affect security and stability and that's exactly what we're here discussing, there's clearly enough input that triggered our activity that indicates that enough things changed so that all by itself just the fact that there was a significant change is important from a stability point of view. And then there's further questions as to whether or not there is a stable resting point that one can reach after all of this and it's appropriate or not and that's sort of more complicated question. But to your question of who made the change, the answer's the operator of the registry, full stop.

Mark Kosters: I think we have a valuable set of data for a 19 days of service was operational. And it would be very interesting for people to actually look at what things have changed and actually present hard data to this committee to say, okay, how did it affect as opposed to seeing things painted in a -- perhaps a subjective manner.

So, I think that going through some of these things in terms of who did what, when where doesn't make a lot of since, we have a love the data to cull from and let's look at this and go forward.

Steve Crocker: I think in fairness if you want to say, well, one might object to the way it was done but nonetheless we have data that we should look at. One consequence that also sets a precedence of running those kinds of experiments. That is not only did the system change how did the processes for making changes change, if you will.

That is does this now encourage or legitimatize or make changes to the system by somebody else running an experiment like this, there are other actors in the system, I mean could make changes and then say, well, we did all of this analysis ourself and public process are a little slow and bulky, so we've decided to move things along more smoothly here a little faster.

One could say Cisco runs most of the router software and decide to usefully improve the operation by issuing upgrade to its software, for example. Or one can pick other large players.

Rusty Lewis: this is Rusty Lewis, if, I could make a comment, Steve to, your point about the data that the committee has amassed at this point in time just to follow up on Mark Kosters' remarks.

That's one of the things we're here today to try to do, we're trying to exchange the data that that we have and get data back from the SECSAC and broader community. In all honesty we've heard a lot of anecdotal data we're trying to get our arms around that data to put that data in some sort of context. As Scott Hollenbeck said the results that we got back from the technical review board were that while there was some impact it was inconsequential. And the service calls that we've handled has been a handful of companies. To the extent you have -- extent that you have hard data that there is substantial or there is even a security and stability risk here, we want to hear that. But we don't want to just presuppose that this service that end users we believe clearly want and needs to be forever buried. That doesn't make sense to us. So we've reached out here today and presented an awful lot of data that we've amassed before we launched the service and after we've launched the service. We have continued to reach out to the SECSAC and asked for the hard quantifiable evidence that you have to refute our position that there are a handful of companies, and as we've said we have customer service reps out there, Chuck Gomes pointed out the fact that we are willing to work with companies, to address these issues. but in the grand scheme of things the supposed stability and security issues that you're alluding to, we just don't see the evidence. So we're here, this is supposed to be a two-way dialogue. we're looking for data from you and we're sharing our data as well.

Steve Crocker: actually it's supposed to be a discussion and query for us to understand what's going on and then speak to ICANN and to the community. The actual flow of information as it should be is for -- when there's a change of this magnitude that it be vetted through sensible process t