SECSAC Meeting
Real-Time Captioning
15 October 2003

Note: The following is the output of the real-time captioning taken during the Security and Stability Advisory Committee meeting 7 October 2003 in Washington, DC. Although the captioning output is largely accurate, in some cases it is incomplete or inaccurate due to inaudible passages or transcription errors. It is posted as an aid to understanding the proceedings at the session, but should not be treated as an authoritative record.

Steve Crocker: Let me welcome everybody. My name is Steve Crocker. I chair ICANN's Security and Stability Advisory Committee. This is the second of two meetings that we have – that we are holding.

I trust that everybody who has come knows why they're here. We're in a very nice facility at the Academy for Educational Development up here on the eighth floor and the path here is tortuous enough that it's its own test, I think. And I appreciate everybody coming and spending yet more time with us. We met a week and a day ago, and had an extended session with presentations from a number of people looking principally at effects with an introductory presentation from Verisign on what they had introduced in the middle of September.

This time we're going to focus primarily on an extended set of presentations from Verisign on not only what they implemented, but the processes that they used in advance and the information that they've acquired since then. We also are very pleased to have two other presentations: one from Ben Edelman at Harvard and one from Global Name Registry which I think will be coming remotely late in the afternoon. I appreciate the effort everybody has put into this, particularly the people who have traveled -- in some cases, traveled again to come here.

Ben, Paul Vixie, I'm sure I'm going to leave people out if I try to do this carefully. And some people, Steven Bellovin called up and is having train trouble and should show up here by and by. Other people's planes didn't comes off. Russ Mundy sends his regrets, and so forth. I'm going to briefly go through a couple of the slides that I used last week just to make sure everybody is on the same page, and introduce a tiny bit of information that we've accumulated, and then turn the floor over to a quite stellar team from Verisign to take us through all the steps here. So just quickly, ICANN's charter covers the Domain Name System and addressing. ICANN strives to achieve coherence, stability and security.

Almost all of the operational details are actually carried out by others. One component of ICANN is the Internet Assigned Numbers Authority that operates within ICANN. Most of the people have been on the committee from the very beginning. This is a committee of outside experts that was chosen for, first of all, their expertise in the Domain Name System and in the addressing system, and, second of all, for their connectivity to operating parts. Everybody on this committee has an ordinary day job. There are no bureaucrats, no attorneys, no lobbyists, or marketing people, for example, on this committee. Everybody has some connection to an actual operational entity. And we also have a fair amount of geographic breadth.

So we have representatives from root servers, from top-level domain registries, from registrars, from address registries, from the security community, and I'm sure I'm leaving out critical components. But a quite distinguished group that has been in place for many months, unrelated to this particular sequence of events. So we haven't shuffled the deck at all.

And I think I've just covered this. Did I leave anybody out? Regional Address Registries are the address registries, the top-level domain include both generic and country code. Our committee is one within the ICANN universe that deals with security and stability. I've listed us as one of the specialist groups, and IANA being the other and there are constituent committees. This is all in the handouts, I'll skip past it.

Within the entire space of folks who worry about different aspects of the operation of the Internet, here's a two-dimensional picture that goes from architecture up through policies and laws, and has as another dimension, the geographic spread. This is illustrative, definitely far, far from complete. So if your favorite organization is left out, hopefully there's a place for that on this chart. The purpose of this is to show that in a limited but very broad away ICANN fits across the geography and multiple layers of this picture. It doesn't actually do the engineering, of course. It doesn't do the architecture, and it doesn't do the law enforcement or policy and laws, but it touches on several layers of protocol and several layers of this hierarchy and covers the entire global structure. In response to the events that we've been talking about, we opened up a comments line, secsac-comments@icann.org, and acquired lots and lots of e-mail. This is the results after taking out the Spam which easily more than doubled the amount of mail. Just one of the chronic ills that plague us all. We had some help from Elias -- I'm going to blow this, Elias Bachrach, and Paul. Is Paul here? Or is he sleeping?

(inaudible).

Steve Crocker: and Mike St. John and Jim Galvin, combing through the mail and trying to assemble the pieces for us.

We received a handful of positive and negative comments and neutral comments, and broke that down into relevant issues, non-relevant issues, and no issues, it was content free. Our goal is to identify specific things that were broken or right or whatever, and some were just emotive. The complaints, we'll document all of this in more detail. This isn't the time and we haven't got all the pieces laid out, but in broad terms, the largest complaints were about Spam filters, about the fact that the change was introduced without lead time, some privacy issues, bandwidth consumption, and a variety of problems with mail bouncing or being delayed. Some secondary complaints about misleading error codes, inability to opt out, and some difficulty in debugging network issues where it just wasn't clear what was going on and sort of added another layer of complexity. Today, as I said, we're going to have a lengthy focus on Verisign. This is a nominal agenda. What we're actually going to do is have a sequence of talks, and then open up the floor for questions. So in all, I think we're looking at approximately two hours, and I think the Verisign folks want about an hour or so of that, and then there will be, as I say, a lengthy set of -- question-and-answer period. As we did last time, we have members of the committee who are participating by phone and web cast. I'll give them the first opportunity to ask questions. I'll go to our committee members who are here physically, back to the phones, and then wide open from the floor. A couple of points. Let's see. I should try to introduce who is here. I should talk about where the bathrooms are, and the protocol for coming to the microphone. Did I miss something, Jim?

Jim Galvin: SECSAC (inaudible) comments for people on the phone who want to --

Steve Crocker: secsac-comments@icann.org.

The other things, let me introduce the members of the committee who are here. Mike Kosters, and Ray Plzak, Ken Silva, and Steven Bellovin will likely come when Amtrak delivers him from AT&T research, and Jim Galvin who is executive support for the committee. And Jim, remind me who is on the phone.

Jim Galvin: actually, we have Doug Barton, Fredericko Neves, Jaap Akkerhuis, KC Claffy, Ram Mohan, and Suzanne Woolf, and Sam Weiler, and Rick Wesson.

Steve Crocker: that was a lengthy set of people. I was going to try to repeat them back but you've exceeded my short-term memory here.

But an equal number, plus, maybe twice as many people as here, so a good number of people including some invited guests. The microphone is here. Please, when question time comes up, please come up, introduce yourself, and before you talk and while you're standing in line, if there's a line, please write down your name. We are transcribing everything. We are absolutely fantastic transcription services here. If you fail to write your name down, you may trip as they catch you trying to leave the microphone and you'll get a second opportunity to write your name down. Lavatories were on your way in, sort of back toward the elevators. There's coffee and soda, I think, outside continuously. Feel free. We'll take a break, but mostly you should feel free to come and go as you wish. There's handouts from last week, handouts from this week, and a bound material from Verisign as well that just arrived.

Any other logistics that I should have covered? So with that, let me turn the floor over to Anthony Renzette, and we'll move -- oh, I'm sorry. Ben is going -- you're going to say an introductory word? So Ben Turner, Vice President of Verisign, will start things off and sort of orchestrate from there. Thank you.

Ben Turner: as Steve said, my name is Ben Turner and I run the name services group of Verisign. and I want to take a couple of minutes and thank ICANN and SECSAC for inviting us to share this information we've gathered over the last several months. We're also very interested, as Steve referred to, as the pieces come together in the SECSAC information collection to receiving that information and taking a look at that.

So what we want to do today is go through, really, four major, five major topic areas. We're going to have Anthony Renzette take everyone through the pretesting we did. We're going to have Scott Hollenbeck review what our technical panel found, we're going to have Matt Larson go through the technical issues and give you our feedback on them. I'm going to go through the consumer feedback on the service. And then Rusty Lewis is going to close it out. So what I'd like to do is have Anthony come up and take you through the pre-launch information. And if at all possible, we'd like to go all the way through the presentation before there are any questions, because I think it will be more complete if we're able to do that.

Anthony Renzette: thank you. So what you see here and what we'll be talking about during this presentation of the larger presentation is the actual activities that we went through prior to developing the actual concept for SiteFinder, prior to developing the product and service that became SiteFinder, as well as the different activities that we did along the way that were extraneous and extra steps we did to ensure the development went smoothly and without the hitch.

The concept of SiteFinder itself came from research we did and in the course of being a registry. In the course of providing our registry services business we routinely conduct research in the market on buying behaviors of domain name holders, people that use domain names, and the types of services that they like to see when purchasing domain names, when purchasing value-added services on top of the domain names and the things they like to see that help them utilize those domain names fully, and then also, of course, renew and continue along the cycle with our customers. The research that we did in this particular case took place in October of 2002. It was designed to, again, assess the needs of consumers and small businesses known as small or home office (SOHO's). It included over 1300 respondents, conducted in on-line interviews representing different demographic areas, and the confidence in this research was a 95% confidence level with an error level of plus or minus 3%. What's interesting about this research is, as I said, the majority of the questions and the research in it are designed to determine buying behavior and likeliness to renew and attitudes and products that affect that. We also asked open-ended questions at the end of this research to assess some of the current pains the Internet users and this type of user have in particular. Of the things that came back in this, one of the top three items that came back was, quote, new ways to find URLs you're attempting to find which drilled down to spell correction on the web. This was the first part that let us on the road to SiteFinder. .

Having this initial concept we of course need to do more research. We realize, like many product development companies, that having initial concept isn't enough. You need to flesh it out, you need to ask the market what they mean by these concepts. More importantly, you need to figure out what types of services would be proven beneficial to the end customer. To do this, in this particular case we conducted two very identical but with different respondent sets, but two identical in methodology sets of research. One in December of 2002, so very soon after getting that introductory feedback, and the second in January of this year. In this case, we floated out the idea, we compared the concept of a spell correction page and response set to the additional error page that people already see. What was very positive is that there was a very high level of awareness. People knew what we were speaking about when we talked about the traditional error response page.

We then floated out a number of different features that could be included on a new response page, and there are two that floated very highly to the top. The first was an ability to initiate search. You can see in December 2002, approximately two-thirds or 67% of users thought that would be highly beneficial, and also links to related or relevant sites accounted for 65% highly useful criteria. In January of 2002, we are encouraged to find that these users now thought that the preference towards search had risen to 70% while links to related sites had risen even higher to 68%. The final take-away from this was that spell correction was, indeed, a feature that users liked, but we also need to include two additional features: of course, the ability to initiate search, and links to related sites and capabilities. So how did we get from this concept to then what we know today as the SiteFinder service? Well, in developing any product or service, we always try to follow a couple of key rules to live by. These are all equally weighted and of equal importance, the first of which is to meet the end-user demand; in this case, to improve web browsing experience as you saw from the original concepts. Second, of course, is any service that we launch, strive fully to be standards compliant. Also in a business of our nature and our size and impact, the service must be scalable. And of course, the service must be able to maintain stability and security of the Internet in general. We, like many other companies, were aware of wildcards that existed out in other registries, we found two that utilize them today in dot tv and dot cc. And we also are knowledgeable of other registries that are using similar work. What we really did here that was new for us, and I think new for the SiteFinder product, was that we took this concept, this spell correction, we added it to a concept to a wildcard in the zone and came up with this, the wildcard was returned when someone mistyped the name, it would include a correction of what the user had mistyped to get them there, as well as the search box functionality and links to related web sites. Of course, as I just mentioned, there are a number of rules that we live by.

Meeting end-user demand was one of them. We also need to take seriously these issues of scale and stability. One of the first things that we need to do is to try to figure out exactly what does that mean, what is the volume, what type of traffic would we see with this service?

Obviously doing the number of transactions that we do, this could be a very large number. The way we do this is that we actually contracted with an external group to provide some statistical help and using that, pulled 30 random samples over a period of seven days. Each of these samples included approximately 3,000 responses. This ranged across the entire dot com, dot net system, which yielded a total of 16 million responses collected. We had a statistician helping us with the analogy, that a 95% confidence level would yield us a plus or minus error response of 5%. We got a really detailed review of what the DNS traffic would like with such a service. Of the approximately 300 billion monthly DNS requests that we receive and process, we would see approximately 600 million monthly name error responses resulting from web browsers. Very important for determining how the system should be built. This also provided insight into the types of requests which we would drill upon further, and it gave us an indication of how the system overall should be built. An interesting take-away from this is, now, looking back on it, that pre-launch analysis is very, very similar to the data that we received during the actual operation of the SiteFinder service. We also then went to another external group to try to get an idea of what the impact would be, the effects would be of a wildcard response to request for nonexistence domains across various applications. This external party was a group of external DNS experts brought in to look across 13 different categories. You can see some of the examples there. 37 different protocols, and 53 different implementations of those protocols, common end user applications. The results of this testing and analysis, which was fairly detailed, produced a recommended course of action on how we should introduce this.

One of the items that they brought to our attention and recommended strongly was the implementation of a stub server in the instance of e-mail, which of course we followed, and also implemented what they termed in their response a, quote, TCP reset option. what they specifically meant by that was requests for non-HTTP and non-SMTP traffic be responded to as follows in the case of TCP, receive connection refused response, UDP, ICMP, port unreachable. Their conclusion doing that analysis and research is that the user response would not change dramatically given this implementation.

This, of course, was not enough for us to be able to proceed. We also needed to do some controlled live testing. In this testing what we were doing was actually giving a wildcard response and a record response instead of a name error in certain very brief and small instances. In this particular case we ran a total of 12 minutes of testing in four different types, receiving a total of 61,000 wildcard responses which yielded 194,000, give or take, hits at our response server that we had set up at the time. You can see that the ratio of response over hits per wildcard response is approximately 3.16 and that had varied somewhere between 3.16 and 5.5. The critical piece of this is it helped us validate some of the earlier research we had done on the protocol analysis, what the spread of those protocols would be, as well as confirming our assumptions on the size and capacity a service of this nature would require. Of course before launching any service of this nature it's also good to get field testing. In this case we were able to work with a diverse range of companies, these were not Verisign subsidiaries. We contacted them through a very complex and externally assisted range of survey and reviews, asking various questions about their reliance on DNS, existing Internet behavior and so forth.

The idea behind this was to identify what responses their production systems would have to a SiteFinder solution.

During the course of this, we contacted over 600 companies. 55 of those companies decided that they wanted to get the full briefing that was interaction with our operations teams, answered questions, and so forth.

And of those 55, 35 decided that this was significant that they need todayed to participated in the testing themselves. This ranged across several industries which was health care, transportation sector, telecom, web crawlers, financial, and software companies. The companies that conducted this testing did it in a number of different ways. One, they used their QA environments and some even used production systems to test these applications against a DNS response server which is configured to give a wildcard response. So their systems in this system would be receiving a wildcard response to a nondomain name query. They also had custom applications and tested custom applications. They tested key applications and some were set to intentionally misconfigure key domain names within their systems to see how the response would be determined. The testing of this nature of course must be based on the responses we received from those companies. We obviously rely strongly on their responses, and the responses we saw from all of them was that they had no major issues to report. There were no impacts on their operations and that the wildcard response did not impact their software applications or their systems.

When developing the product further, so we've progressed now from the concept that it was taking it through the end-user research, taking it further through the analysis on how the service would be responded to, and now we're actually in the product development phase. We understood that the impacts and the responses received would be varied, and that wherever possible, documentation on the methodology followed would be extremely important. This set of documentation is of course available and has been available on the SiteFinder and Verisign sites for quite some time, but specifically, we have a white paper that discusses the implementation of DNS wildcards in general. In specific, we have an implementation guide that covers Verisign SiteFinder, how we followed the white paper, the steps that we took. Additionally there's an application developers guide that was developed by our team in conjunction with others looking at how people who develop software in systems that externally rely on DNS could work with a wildcard in a SiteFinder response. We also, of course, use a combination again of external and internal resources when developing the product itself, and more importantly, when testing and reviewing our processes and procedures. This included external QA or quality assurance testing that was brought in, in addition to our own internal teams to make sure that the steps that they followed met and were as objective as possible. And then we also took our product development processes, our QA processes, our pre-launch processes, and those systems and procedures for another review by an external company familiar with software and Internet software development, and made sure that they reviewed how we were doing this, helped us make modifications and ensured completeness. And then in addition, as Scott Hollenbeck has mentioned in a previous statement, the product itself was released with a monitoring program built in so that we could measure any impacts in responses that we saw once it was live. with that, I'd like to turn it over to Scott Hollenbeck to review the technical review panel that we set up.

Scott Hollenbeck: okay. thank you, Anthony, and thank you, Steve.

Thank you for the opportunity to address the committee this afternoon.

As Anthony said, my name is Scott Hollenbeck. I'm here to tell you a little bit about our technical review panel and to describe the outputs of that panel. overview of the presentation in front of you.

First, the purpose of the panel itself. Soon after the launch of SiteFinder, Verisign announced we were going to form a technical review panel of outside experts to specifically solicit and gather technical information and data regarding the implementation of SiteFinder. We were looking primarily to obtain some objective, sane analysis of a lot of the commentary that was being made in various public forums that we were receiving at our customer service department and at various public e-mail addresses we made available. Stage two was to distill the received information and data into specific implementation issues. After we identified the issues, trying to determine which issues were based on fact concerning the service, and which were based on emotional or political considerations. For each issue associated with the service, determine the likelihood of the issue arising for Internet users and then the consequences of the issue for users. And then based on the analysis of the issues, determine what enhancements, if any, could be made to improve the service, or to provide a work-around for people who are impacted. And then finally, the panel took it upon themselves to report the observed implementation issues to Verisign along with any data supporting the issues, and then Verisign providing the final analysis, ran it by the panel for their approval, and you'll actually see the raw data provided by the panel at the end of this presentation.

So who was the panel? We tried to cast a very wide net to collect experts in various industries, specifically some related with Spam detection and prevention, that's Ken Schneider, the CTO and VP of operations at Brightmail, George Sherman is the CTO of Morgan Stanley with a strong background in Information Technology, Keith Teare, Chairman and President and CEO of Santa Cruz Networks. Bruce Tonkin, CTO of Melbourne IT, and we also have three other members of the panel who wished to remain nameless because they felt they could be more effective if they weren't subject to some criticisms from people floating around in the community at the time. We also had four senior Verisign engineers participating on the panel, Leslie Daigle, myself, Mark Kosters and Matt Larson. Bruce set the agenda and we were there to make sure the panel had accurate information from a Verisign perspective.

So what did the panel do? We tried to look at SiteFinder from three different angles: one was an analysis of all the reported issues that we could find from the various sources I described earlier. Two, we tried to look at SiteFinder from analysis of the protocols that were involved in contacting the response server. Three, we tried to look at what people sitting in chairs were actually doing when they experienced or somehow dealt with SiteFinder, the service. And as far as the sources go, we looked at the statement produced by the IAB and issues presented in other forums, such as the NANOG mailing list, Slashdot, the on-line press, and frankly any other place where we could find sane national commentary.

Okay, analysis of issues. This isn't a complete list of the issues. You can find a complete list in the tables provided at the end of the presentation. But what we tried to talk about here today are those that the panel categorized as having at least a moderate impact on a person. And those are as follows: first that SiteFinder currently provides an English-only web page. And the panel's analysis was that this was an issue that could be addressed by the service operator. End-user error reporting. The panel felt that this was an issue that required software update on some person's part, or some company's part. Spam filtering could be addressed but it required a filter update to do so. Automated HTTP tools could be addressed via a software update. Resolvers that have non-DNS fallback mechanisms such as on Microsoft platforms where if DNS fails you might do a NetBIOS query. Again, the panel felt this was an issue that could be resolved with a software update. Another moderate impact area was the way some people checked domain name availability for registration capabilities. Again, moderate because DNS would be required to change the existing behavior. And finally, issues associated with e-mail delivery, most of which could be addressed by the service operator. From a protocol analysis perspective, the panel looked specifically at the top ten protocols by number of connection attempts. now, this table -- I'm sorry, this pie chart is actually simply a graphic representation of the table I presented at the last review session. As you can see, HTTP and SMTP traffic account for 85% of the traffic we see at SiteFinder.

The panel felt the impact on other protocols was typically a different error message the user would see and/or a slight delay that would be experienced when compared to the pre-SiteFinder experience. The most significant issue from the panel's perspective was that applications don't treat TCP and UDP errors the same way they treat DNS name errors.

So what did we take out of all of this? First, that SiteFinder did not introduce any catastrophic problems. Virtually all of the issues that the panel looked at were categorized as having a minor impact with an unlikely possibility of occurring, but I need to say that that does not mean that they could not occur or that there was no impact. The panel specifically did not identify any issues that had a security or stability issue. The panel did stress the desirability of providing time to adapt and to educate for issues that can't be addressed by the TLD operator. Most of the issues were deemed to be minor or an inconvenience. However, there were some that I described earlier, issues that were categorized as moderate because they would require software change or some proactive change by people outside of the TLD operator. And the next couple of slides, which I won't go into in detail, there's just far too many words here and too difficult to see from the audience but these are the raw outputs from the panel that Verisign's analysis of the discussion that took place. So what you see here is the protocol analysis table, the application use-case table, and the issues table. And in some places where you see empty cells, that's because panel couldn't reach agreement on what the issue was or what the mitigation strategy might be.

That's everything I had. Thank you again for the time, and I'd like to introduce Matt Larson who will actually take you through the discussion of the specific issues that the panel deemed to have moderate impacts.

Matt Larson: hello everyone, thank you, Scott.

So, as Scott said what I'm going to do is walk through what were in the TRP's opinion -- the technical review panel's opinion the most significant issues. And for each issue what I'd like to do is identify and offer a little more explanation than Scott was able to do in the time that he had. Then I'd like to present some applicable data that we have and then finally provide our analysis and our response. So, once again these are the issues, this is the same list that you saw a few moments ago in Scott's presentation.

So the first of these is the issue that the SiteFinder response page was available only in English. And the is that while this is only English previously prior to SiteFinder the browser's error page would potentially have been localized and available in a language understandable by the user. So, Verisign has always planned to introduce a localized version of SiteFinder that just didn't happen with the initial release. So, future releases will include support for at least German, Japanese, Spanish, French, Chinese and other language as our analysis shows that that is what people who are using the site would prefer. The plan is to use the HTTP accept language header display the language and also what many sites, websites allow users to change the language once the -- users to allow the page to display if they're not able to understand the language.

The second of these issues is end-user error reporting. The issue here is that the application behavior in the case of failure changes. And we believe this is at base a user interface issues, the application still fails but potentially with the different error message to the user. So prior to SiteFinder, prior to the wildcard a record and dot com and dot net user attempting a record with application would that have seen. But prior to SiteFinder they would that application would attempt to make a connection and then would see some sort of connection. Behavior change for the users. So, to put in this connection there's no change in application behavior for for domain names. any computer using domain names that work prior to SiteFinder and wildcard also worked afterward. I guess the -- I think the take away here is that a failure is still a failure although we do acknowledge that there's potentially increased user confusion and increased difficulty troubleshooting. For future applications it is possible that application could check for the presence of a wildcard a record on a zone, it could then record to detect synthesized responses and take appropriate action and display appropriate message. I guess one possibility that we have sort of been discussing internally and would like to begin talking about in the community is the potential for a DNS protocol change to indicate a synthesized response. So that a resolver could understand that the response it received was the result of a wildcard and pass that to a wildcard and pass that back to application.

This is a fundamental user interface doesn't affect stability of security. One issue that has gotten a lot of attention it was identified by the technical review panel that was Spam filtering based on domain name existence checks was affected by the existence of the wildcard. And our analysis and reports that we've received from third parties indicate that this issue is more complicated than it might appear at first glance, and that it's perhaps less significant than at sometimes been reported. So, I guess we feel that the reality is that using domain existence checking to identify Spam is first off slow and resource intensive. There's at least one popular mail transport agent whose author believes that this check is not even worth implementing. Feels quite strongly about it.

This economic is also not the obvious straight forward test that it might appear to be. I have more on that in a subsequent slide.

It's also not effective based on our research against a large percentage of Spam. and ideally which I think has gotten lost in some of the discussion of this it really is only one test. Total anti-Spam solution.

I think anyone who is even had a passing acquaintance with the problem of filtering Spam realizes that there's no magic bullet, there's no single test that's going to prevent all Spam. And to that end this is really only the first of possible checks, this is one low-level check.

So, just to be clear, the amount of Spam on the Internet did not increase as a result of the presentation of the wildcard. Let me tell you a little bit about our analysis. We took a look at a very large corpus of Spam. It was over 73,000 messages. And only 3% of those messages contained a nonexistent domain name in the from header. This check can be done different ways and means different things to different people. The first time we went through we did this using an NS query against the dot com and dot net servers. We looked at the domain name in the corpus header. We pared that down to something dot com and something dot net. We did a query to that's where our 3% number comes from. One of the members of the technical review panel took a look at SpamAssassin -- it's Bayesian filters one of the large corpus that is 70% Spam and 30% legitimate mail. On the slide the stats that resulted from that from checking using SpamAssassin's particular check for nonexistence that number is slightly higher, 4.6% of all the Spam in the corpus had a nonexistent domain in the from header. That's this where -- no where near the double digits reported. We just have a hard time reconciling the analysis that we've done in red with some of the other reports that we've seen on this particular issue. So as I mentioned a moment ago, what this has made us realize in the process of doing our own testing is that the whole idea of doing domain existence checking for Spam filtering is a whole -- very subtle process. There's no standard way to do this. And implementations vary in how they do this check. So, for example, gethostbyname in the byname library is not the best way to do this. It only queries for A or quad-A records. In looking at the code for some implementations what they have seen is some Spam filters use this method then don't differentiate between rcode 3, which is the name error or a NXDOMAIN error and no data or no such data error. And what this means is, if an implementation checks for domain existence in this way, they would not be able to tell a domain that didn't exist versus one that has an MX record which could receive mail but not an A record so gethostbyname could have failed.

Unless the application goes out of its way to check the return value from gethostbyname carefully I think what we found is that it's very easy to have false positives to reject mail, that really could be legitimate but this initial filter would cause it to be flagged as Spam.

So, having discovered this we took one of those Spam corpuses and ran it through looking only -- ran it through the same checks, this time using gethostbyname, and purposefully only use gethostbyname at a high level, didn't return success was the structure populated or did we get a null structure, we didn't check the response code and in this case we found that 17% of the Spam messages were identified. So, we think this could be the source of some of the different numbers that we're seeing. That there could be false positives as I said a moment ago. Not in this case right here, because we were looking at Spam exclusively but if you're using this method it's quite possible since there's certainly cases where a domain name has an MX record but no A record, that you would use this test using gethostbyname in this manner and identify it as Spam.

All that being said, I think another thing that's important to point out is that there are people who do feel that this is a check that they want to have as part of their anti-Spam arsenal. and to that end what we saw is that within days and sometimes within even hours of the addition of the wildcard A record in dot com and dot net in various mail transport agents that would issue patches that would allow them to do this check with the presence of the wildcard A record in dot com and dot net.

Another issue was various automated process, in other words not human at a web browser using HTTP over TCP port 80 that think could exhibit problems when encountering the SiteFinder page. Instead of nonexistent domain in causing abort connection couldn't take place, they would attempt to connect to the SiteFinder page, see the SiteFinder response page and then potentially be confused. We have not heard any reported occurrences of this, so we're not able to offer any specific data. No one has told us of any application that was in this situation. In the event of the process that specifically crawl the web, web crawlers and other spidering tools like that, the site does include a robots.txt file which would prevent them indexing the site. I guess more basically using HTTP over port 80 in this way by automated tool is discouraged according to BCP 56. The next issue identified by the technical review panel was name resolution process that continue with other methods, if DNS fails. So Scott mentioned this briefly a moment ago, some versions of Microsoft operating systems and may very well be others that could be configured in this way, if they use multiple methods to do name resolutions, such as DNS and another method like NetBIOS and if they're configured to use DNS first and if they're configured to stop when they get a successful response it's possible that the presence of the wildcard would then cause them to believe that name resolution had succeeded they wouldn't continue on with another method such as NetBIOS.

So, in some cases case a workaround is available, some cases revolve changes the order in which the methods are used and use DNS methods last. We are aware of some configurations are using and intentionally to force resolution to the next method. So, for example, using such a name in a resolve or search list that essentially forced every name resolution attempt to try DNS, to fail because of the intentionally nonexistent domain then proceed onward to the next method. and building configuration that relies on the nonexistence of a domain name like this, is really very dangerous and unwise. Particularly given in the case of dot com and dot net any one can register any domain name they like. That name could be registered out from under someone. And if anyone has looked at the density of names in the dot com and dot net zones it's amazing the kinds of names that are registered, it can be nontrivial to pick a nonexistent domain even one that you would think ought not to be there really, really is. And additionally RFC 2606 does define top level domains and sample dot com and dot net domain names.

For example, it could be used to construct nonexistent domain name that could be used for this purpose. Some applications use a DNS address record query to check for domain name availability and they would potentially be affected by the presence of the wildcard a record. Now, I think it's important to point out that there are various names that don't get put in the dot com and dot net. For example, reserve names, names that are on hold, and domain names that are registered but don't have any names specified. None of these mix it into the zone, there for any kind of a DNS query for them would not detect them they would be potentially viewed as available when in reality they're registered and unavailable. So using DNS for this purpose is not recommended.

Registrars that want to perform this check can use the RRP check command and the public can use WHOIS. Now, there were three issues related to e-mail specifically that the IAB brought up in its response that the technical review -- these were delivery to nonexistent dot com or net domains that now required additional processing to contact the SMTP bounce server. Also, an issue that the IAB related with misconfigured MX records causing a situation that had previously been a soft failure that would not have caused a message to bounce in the presence of the SMTP bounce server turning into a hard failure. And then finally mail clients or mail user agents with misconfigured SMTP servers that attempt to submit mail to the bounce server got a misleading -- potentially a misleading error message. So, we took all this feedback very seriously and thought very hard internally about ways that we could -- ways that we could work around this and also the community presented some valuable suggestions and one of the things that we are considering is a change that we think addresses these issues in really I think a pretty good way. That is the addition of a wildcard MX record with a nonexistent target domain name to the dot com and dot net zones. For example, as you see there. And just to illustrate my previous point, the first time I put this in the slide I had the domain name on the right here as doesnotexist.com then I thought, I better check make sure that that is not in fact registered before I put it up on the slide in front of a room full of people. And sure enough someone had registered doesnotexist.com I had to find a different name. It would be a wildcard MX record whose target mail exchange entry simply does not resolve. Along with this we would stop the SMTP bounce server, would not run it any more and connections to TCP port 25 would be reset. So, looking at these issues in the context of that potential solution, I'd like to go over what our initial analysis shows and I guess make a request for additional feedback on this from the community. In the case of delivery to nonexistent dot com or dot net domain names a server that is compliant with server 2821-compliant server, which is latest and greatest RFC is going to query for MX records, going to receive the synthesized response resulting from the wildcard record, and according to RFC 2821 it's supposed to report an error when all the MX records are unusable which they would be, because the target would be nonexistent.

So, in our testing we found that recent versions of the most popular mail transport agents the list here, Sendmail, Exim, Courier, qmail and no particular order, they all treat that as hard failure hand bounce the message back to the sender. That would be the desired behavior. Postfix does treat this as soft failure and requeues the message. The advantage of the record in this situation is it does move the processing back to the SMTP client and it eliminates any dependencies on the SMTP bounce server. So, therefore, any -- the performance of the bounce serve we're have no bearing on the solution because it's not required any more and the solution, others had mentioned possible privacy concerns about the bounce server and although we said very clearly that we don't even have log files there to collect any information and our privacy policy specifically mentions this, the situation completely goes away because the bounce server would not exist any more. The next issue was misconfigured MX records. Just to restate this issue was the situation where a domain would have multiple MX records, one of which would have a typo that would cause it to be a nonexistent domain, however in the presence of the wildcard that MX record would not be rejected by a mail server, a sending mail server now the MX record rather than having a nonexistent target now becomes usable and the mail server would attempt delivery to the bounce server. And the bounce server then would respond with a 5xx error indicating that the message couldn't be delivered and as I mentioned a couple of slides ago, this turn what had previously been a transient failure into a permanent failure cause the message to bounce. So, after this issue was reported we took a look at -- looked at all the dot com and MX records. We did a query for every single registered entry at dot com and net. The thing that we took away this section is extremely rare condition compared to some other situations.

So just to give you some data we thought was interesting and worth sharing, the number of MX records that lead to known unroutable addresses that is when you resolve the target of the MX record and look at the address the numbers that are -- the number that have known unroutable addresses is just over 6%. The number of MX records that have IP addresses rather than domain names as target a common error by people who don't understand the proper syntax of MX records that came in at 1.5% of total MX records. And then the number of MX records whose targets were nonexistent domain names in dot com or dot net was 0.7% that is roughly 2,000 times less than 1.5% number there. But in any case, with the elimination of the SMTP bounce server, we believe this would no longer be an issue because now misconfigured MX records become unusable. Although in the presence of the wildcard record, the target of the MX records would still resolve, there would be no bounce server there to accept any connection, the connection would be refused. So, we have only started to do some testing on this. But it is our initial presumption that mail transport agents react more favorably in this situation, namely a reset connection than they would to some other 5xx response. Let me say that one of the other solutions that we heard from the community was why don't you keep the bounce server but just run in such a way that it returns SMTP 554 which is the way within the SMTP protocol to formally reject projections. We looked at that and also looked at the wildcard MX solution. Based on our preliminary analysis the solution I'm describing to you here, namely wildcard MX record with no SMTP bounce server running at least based on our initial analysis seems to be the best solution. Then the final e-mail related issue identified was that just to recap that, a mail client would a misconfigured outgoing SMTP server that previously would have been nonexistent in the presence of the wildcard A record would resolve that mail client would have submitted mail to the bounce server would have gotten potentially misleading error message saying that the domain doesn't exist. With this solution that I've been describing, the bounce server isn't running any more so that mail client would get a connection refused when it tried to submit mail to its misconfigured out going SMTP server. And again it's our presumption that based on initial testing anyway that this situation is more favorable than a bounce server that would return 554. So, that's my presentation and I will turn it over to Ben.

Ben Turner: what I'm going to go through is how consumers reacted to the service post-launch. So what we did is we went out and we fielded three different sets of surveys, or research. we did one in the U.S. where we selected a random sample of 1,027 consumers.

That sample set had a plus or minus 3% margin. The vendor we used to do the research is a company called Markitecture. It does a research for a lot of the leading consumer package companies for different technology companies. The second society of research surveys we did was yet another again segment of the U.S., 1,000 Internet users randomly selected that had margin of error of plus or minus 5%. The company we used to do that was Customer Strategies. And then the final set we did were international we did one in the U.K., one in Germany and one in China, each one of those -- the users with the plus or minus --

John Crain: can the person if you can hear me put us on hold? Take it off, please? You probably can.

Ben Turner: so, the 300 end users in each of those markets were surveyed that gives you a sampling error of plus or minus 3%.

So, what did we find out from this research?

Basically what we found out is that there was strong preference for the SiteFinder error page versus the current error page. The U.S. 84% had preference in the U.K., it was 81%. in Germany 61%. In china it was 76%.

The other interesting column to look here or row to look at here is the last one which is strongly preferred the current error page, that you can see the U.S. 5%, 6% in the U.K., 15 in Germany and 10 in China. So as we went deeper in understanding the research, the other thing that we found is that 76% of Internet users rated the page excellent or very good. So the way to look at this chart is, is the column on the left gives the total view of the site or the SiteFinder page where you see the 76%. Then each of the rows or columns to the right gives you each of the areas. The search box 34% that use search box rated that.

Excellent or very good. Web suggestions or the digi-spell correction 38% and popular 45%. The two key numbers here, the take-away numbers are the summary numbers which is 76% rated the site excellent or very good.

And 4% rated it poor. The next finding out of this was that over 60% found the site easy, convenient and useful. And again this breaks into the different areas, the different areas on the site. On the far right the two columns show the difference between light and heavy Internet users. so on the light side 59% found it was easy, convenient, useful.

While 67% of the heavy Internet users found it easy, convenient and useful. We found to be an interesting differentiation. So the other thing we found is that over 50% said it makes the Internet better, while 3% said it did not at all. So if you take a look again at this chart it's laid out exactly the same way the far left column gives you the summary. The 53% shows you the extremely and very well. Bottom row does not describe at all they thought it did not improve the Internet at all. I'm not going to read to you completely but this gives you a sense of some of the direct feedback or the testimonials we got from doing the survey. The top one comes from a guy at Mercy Behavioral Health.

And then basically what that says is that there's more functionality than you get with a 404 so it's helpful for me. And what I'd like to do now is turn it over to Rusty who will wrap things up.

Rusty Lewis: thank you, Ben. And good afternoon everyone. And thank you to Steve Crocker and the SECSAC and ICANN for giving us an opportunity to present our data regarding the SiteFinder service.

We're happy to be here today because it's an important conversation we're having. As is the case with any new service the provider has to listen to the marketplace to be successful. We spend a lot of time today talking very openly about the issues we've encountered since launching this service. We've had a lot of constructive input from many people in the technical community. And we're very grateful for that. Believe me, we want to assure you that we are listening, we do care, your input is important to us and we plan to act upon it. What I'd like to do is just share some of the next steps that we're considering at the present time.

If and when we relaunch the service, we've gotten several specific action items that we're considering and we'd like to solicit further input. Many of these issues, Matt and Scott have covered in their presentations.

First, response to a number of requests that we've received for advanced notice and that was also reinforced I think in Steve's opening comments as well as in the technical review board comments that Scott Hollenbeck covered. We've believe that it is appropriate to give the community advance notice at least 30 to 60 days before we would re-launch the SiteFinder service. Now, I think it's worth noting that many thought before we launched the service that the presence of wildcard in the DNS would somehow break the Internet. Well clearly the evidence does not support that at all quite to the contrary, properly configured applications do work, our experience when the service was live that we were able to continue to process over ten billion queries each and every day with 100% availability in the com/net zones. The number one complaint actually that we heard across the board was this lack of notice. And we do sincerely regret any inconvenience that the service may have created.

And notwithstanding the fact that most of the concerns that we've discussed today and the ones that we're aware of were largely addressed in a matter of days if not hours, we do believe that a longer notice period would be appropriate. Second, as Matt Larson went into some detail we do believe the addition of a wildcard MX record would address many of the misconfiguration issues that crop up, in e-mail applications, as well as some of the privacy concerns much we did make it clear that we don't retain or collect any data, but the existence of a wildcard MX record would put that to bed once and for all for the privacy community. Third, we talked about the needs of non-English users.

And we do feel that localizing the version of SiteFinder for the international community is certainly an enhancement to the service that's well worth doing and it is one as I believe Scott or Anthony had mentioned in their presentation that we do have plans on implementing localized versions of the SiteFinder service. And finally, based on all of the input that we've received since we've launched the service, we do want to go back and update our white paper and we'll be looking for feedback from the broader community over the next few weeks. So, the bottom line is we're very open to suggestions from the community, we are going to continue a broad out reach effort to get additional comments and we welcome your thoughts. I also think that it's important that we sort out how standard compliance services like SiteFinder can be launched. From our perspective, we struggle with the notion of what's the point of standards, what's the point of best practices, if those who choose to ignore standards are favored over those who follow the rules.

Clearly, we know other registries are interested in exploring similar services, and we believe it's important to develop best practices to promote and encourage uniformity and consistency. A stable, a secure, an interoperable Internet we believe is, in large part, dependent on standards and best practices. Finally, we believe that encouraging innovation at the core is just as important as encouraging innovation at the edge. If that is not encouraged, it's going to result in less investment, less R&D dollars that go into the Internet infrastructure and we believe, ultimately, a weaker Internet. We believe that's a problem that should concern not only registry operators but policymakers, the technical community, and all of us that are worried about the growth and prosperity of the Internet. So I'd like to again thank the SECSAC and ICANN for the opportunity to present our supporting data on this service. We look forward, Steve, to reviewing any additional information that your committee may receive. And also we're looking for additional feedback from the community to many of the thoughts and suggestions that Matt Larson has raised in his presentation. So with that, let me turn it back to Steve for questions and answers, and I'd ask Ben Turner to come up who will serve as our MC for the Q and A section of this presentation?

Steve Crocker: thank you, Rusty.

All right. Let's see. It's now 2:20, and the agenda -- the schedule that I put up before is no longer operative, as we used to say. Let me ask you guys who are here, do you want to take a break now or do you want to roll right into questions and take a break later? Quick show of hands?

>>>: break.

Steve Crocker: yes, raise your hand if you want to take a break now.

Too bad.

(laughter).

Steve Crocker: okay. I think, then, we are primed here and ready to go. As I had said, wasn't to ask people on the phone first, come back to the committee, then back to the phones and then we'll take questions from the people who are sitting here patiently.

All right. Anybody on the phone?

Ram Mohan: Steve, this is Ram. I have a question.

Steve Crocker: go ahead.

Ram Mohan: the question is why did the SMTP problem not show itself during your initial tests, since it seemed to be a basic error that would have shown itself early and often?

Scott Hollenbeck: Ram, Scott Hollenbeck here.

One of the things we found were that a large number of the error situations were typically due to misconfigurations and as Matt said in his presentation, the actual incidence of the misconfigurations is actually fairly rare. So while we did anticipate some misconfigurations, a lot of them were due to situations that we didn't become aware of until they were brought to our attention post launch.

Steve Crocker: any follow-up? I'll ask a question just to follow-up on that.

One of the things that becomes clear from the sequence of the presentations is that you designed a series of experiments, test, research internally and went out and executed that beforehand in contrast to the more public kinds of processes where things are designed and tested externally, shared among the community. Do you have any numbers on how much your testing did compared to the quality of the testing that takes place by those public processes? That is, you have 95% confidence levels for, you know, so many thousands of responses. Do you have comparable numbers for what the depth and breadth of testing that takes place by the usual public processes?

Ben Turner: I'm not sure there are usual public processes for testing. What we did is we followed, you know, one version of best practices to test, and we're fairly confident in the way we did that testing. It's no different than the way Verisign tests the majority of the products that we roll out.

Steve Crocker: anybody else on the phone want to follow up on that or anything else?

Ram Mohan: Steve, this is Ram again. I have a different question.

Steve Crocker: go ahead.

Ram Mohan: other than the user interface surveys and tests, what other non-web-based tests were conducted? That's one question. And the second is what kind of testing was done for user applications as compared to user interface.

Matt Larson: this is Matt Larson, I can address that as a stall for time moving quickly back through the presentation. One of the things Anthony touched on was this, and I'm looking for it.

Here we go. We had a third party, and the slides aren't numbered it's from Anthony Renzette's presentation, the title is SiteFinder Development Research, Third-party Testing. And we engaged a third party to do some testing, and the numbers are on the slide here. We had them test -- identify different application protocols, the most popular ones.

Actually, they identified the list. We worked with them to do that, and then they tested, it was 53 implementations of these 37 different protocols that fell into 13 different categories. And they did that testing to evaluate the impact that a wildcard would have as well as these were all network clients, so what the behavior would be when those clients connected to the wildcard address and got either a TCP connection reset or a UDP port unreachable. And their findings was that the user experience would not change dramatically.

Steve Crocker: any other questions from the phone?

Let me move to the panel, and then -- anybody on the committee want to ask questions here?

David Conrad: this is David Conrad. A quick question about the survey itself.

Steve Crocker: speak up a bit, David.

David Conrad: I'm sorry? Are there any plans to actually survey against -- instead of the error page versus SiteFinder, the SiteFinder versus AOL Search or MSN Search?

Ben Turner: so when we did the survey, we caught all the varieties of error pages that exist, so of the existing error pages, what was your view of SiteFinder versus the existing. That would cover MSN, AOL, the whole variety.

Steve Crocker: let me follow-up on that. Do you plan to make this data available? I mean, you presented summary information here, but the experimental design and the instrument you used to do the survey and the responses and so forth?

Ben Turner: the end-user surveys that we did, the results we just took you through?

Steve Crocker: yes.

Ben Turner: what we can share with you is what we walked through. We can give you the additional tables, if that's helpful.

The actual, you know, feedback that we got directly from doing the surveys, you know, is proprietary information to the company, and just like most companies, we're not comfortable sharing that in an open forum where we have competitors involved in the process.

Steven Bellovin: what the questions themselves? The wording of the questions are very critical in the answers.

Ben Turner: I think what we have believe we've done is a very methodical set of surveys. We used third-party researchers to do that that are well qualified, and I'm not quite sure what concern you're getting at there.

Steve Crocker: let me jump in.

Within a stone's throw of where we're sitting is heart research well-known pollster. I hung out there just to see how things are done.

It's eye opening. Polling is a proactive, not a passive operation, and the structure of the survey is all important. And many times, and I have no idea whether it applies here, but many times, polling instruments are put together for the purpose of conveying a message and what's called push-pulling as opposed to getting a neutral answer.

There's a very, very big range of results you get depending on whether you're putting together a carefully engineered, carefully designed experiment in an academic setting versus a market-oriented mechanism.

There's no way to tell from what you've presented here, unless that gets opened up and examined.

Ben Turner: there actually is a way to tell, and let me show you.

So the way to tell is you see multiple responses within a question. You see multiple samples. You see multiple responses within a question, you see multiple questions. What you're talking about, I could go out and design a one- or two-question survey and get an answer, but the fact that we designed a survey or market research company designed a survey that had multiple questions with multiple answers for each question eliminates the concern you're talking about. In addition, by having a truly random sample, you're not going to be able to stack the deck, if that's what you want to call it.

Steven Bellovin: it's not even a matter of stacking the deck. It's a matter of what is it you're measuring? That's why I want to see what the questions are. It's a very subtle business.

Rusty Lewis: this is Rusty Lewis. if I might just address this issue.

As I mentioned in my closing remarks, the success of any service really depends on what the reviews are from the market. And from our perspective, we did not conduct this survey for purposes of convincing the SECSAC or ICANN or anybody else, for that matter, that end users want to use this service. We're a commercial company. We are interested in getting unbiased, unfettered feedback from the marketplace so that we can design services that meet commercial needs. I mean, so clearly we're presenting here data after the fact that we did before the fact to convince ourselves that it made sense to roll this kind of service out, and that, in fact, it would be useful to the end users. So I can assure you that from the standpoint of the way in which we went about hiring the best people we possibly could and to get the kind of information, Steve, that you're getting at, we want real feedback. Just like in this whole process, I mentioned the fact that we've gotten a lot of constructive feedback from the technical community on ways to improve the service. We are sincerely grateful for that. We want to make the service work for the end users, and that's what we think is the value of this community input.

Steve Crocker: and so, it's fine. Did you ask the community of prospective -- of users who are going to be affected in what ways will this break your existing applications or interfere with your existing work? Sort of the complementary question to the ones of "do you think you would like to see a search service?"

Ben Turner: I think if you look, to give you a sense, we would not ask the question if we did not want to have a fair, balanced, you know, understanding of what users really thought. "Do you think this is poor?" this clearly shows we're asking those types of questions.

Steve Crocker: okay. We should move on to other things.

Anybody else on the committee? Ray.

Ray Plzak: I want to go back to this right here. You're asking people about the web page itself, the usability of the service once they have arrived. And I could do a similar thing to other search services and probably get a very similar set of responses.

And so I'm kind of curious about the usability of this web page, how is it germane to the other things that we've been talking about here in terms of DNS and SMTP?

Ben Turner: I think Verisign's view is the Internet is made up of multiple user sets. And one of those user sets are consumers who actually use the Internet to surf, to buy things, to send e-mails.

That's what we were testing with this survey.

If you guys don't think consumers are relevant, we can move past this data set.

Ray Plzak: I didn't say I thought consumers were irrelevant but I'm saying I could go to a similar service, like Yahoo! or whoever you want to name and conduct a similar survey and get similar responses.

Ben Turner: I honestly don't know if you did this survey on another web site what kind of answers you would get. You might or might not get the same answers. I have no idea.

What we tested here was did you find this site useful and did you prefer it over your existing error pages.

Ray Plzak: and what does this have to do with the effect on DNS and SMTP?

Ben Turner: well, like I said, this is targeted towards a specific set of segment that's out there that's using the Internet.

This is the end user, the browser user.

We presented other information that covered the other protocols.

Ram Mohan: this is Ram. Would it be appropriate to say that the survey was fundamentally focused on web users of the Internet?

Ben Turner: the survey we're talking about, that is correct.

Chuck Gomes: and Ram and Ray in particular, I want to respond in particular to your question. My name is Chuck Gomes with Verisign.

One of the segments of the community that has not been looked at in this whole issue, in my opinion, is the user community. I believe they're very relevant. Now, they're not the only user, but they are very relevant. So like Ben said, if we don't care about that part of the community, we should move on. But I think everybody does.

Ben Turner: the other thing, Ram, is we did do a separate set of research where we contacted over 600 enterprises that would not be web browser users; right? So there are clearly multiple segments here that have to be addressed.

Steve Crocker: other questions?

David Conrad: I just wanted to try again because my question wasn't answered.

Specifically, is there any plan for Verisign to do a survey that compares apples to apples, that is the user's experience with SiteFinder versus the user's experience with AOL or MSN search that result from a non-existent domain lookup attempt?

Ben Turner: sorry if I wasn't clear. We did do that research. We went out and asked the question, you know, vis-a-vis the current error page you receive, and we found four major -- five major types of error pages that people saw, you know, what's your views versus those versus SiteFinder. And that's what this data shows, where 84% of the U.S. users, 81% in the U.K., 61 percent in Germany, 76% in China either strongly prefer or somewhat prefer SiteFinder over the current error messages they were receiving.

David Conrad: of those results, do you have a breakdown of the ones who are actually comparing against AOL or MSN or whatever searches versus an error page?

Ben Turner: no, all we did was qualify to make sure they had seen one of those error pages in addition to a SiteFinder page.

Steve Crocker: but you are saying that this data is only for people who have seen somebody else's -- that's not what the title of the slide is. It says SiteFinder page versus an error page versus a SiteFinder page versus a comparable service, AOL or --

Ben Turner: we're liberally using the same error page, the error page covered the 404 error page, Yahoo! error page, Google error page, MSN, AOL.

Steve Crocker: that's the question. You included the error page as to what that would look like as opposed to people seeing only the SiteFinder error page versus the comparable results from AOL or --

Ben Turner: the results would not change very much. Most people are seeing an alternative search error page.

Steve Crocker: so you're asserting that most people don't see 404 today?

Ben Turner: what we're saying is the way we did this survey is we tried to narrow it down so the majority of them saw a comparable search page. Error page is probably what's causing the confusion.

Steve Crocker: right.

Ben Turner: we're using that in a market research term, not in a technical sense.

Steve Crocker: right.

David Conrad: however, in the testimonial and verbatim, the very first ones said something about a 404 error.

Ben Turner: right. So we didn't completely eliminate the 404 errors.

The ones we did of a thousand users, customer surveys -- customer surveys to do, we asked them specifically to give customer testimonials. The data you're looking at that shows the preference here is based off the Markitecture research where we narrowed out the 404 error page.

Steve Crocker: thank you. Let me move to a general open forum and allow people who are here to come to the microphone and ask questions.

And it's also been suggested that I announce once again the e-mail address for comments to come in which we're monitoring in real time.

secsac-comment@icann.org. That's secsac-comment@icann.org. Thanks.

Marilyn.

Marilyn Cade: I have two questions. I'm looking at the objective which says this is to address the needs of consumers and SOHO's in Europe and the U.S. You undertook this research as a registry; is that right?

Ben Turner: there's a separate set of research that was done before we launched the service. So the research we just talked about was post launch.

We also, just like we do with any product in Verisign, do pre-launch research, and we did go out, and I think you're talking about -- we did go out and do, as a standard course of business, you know, the naming and directory services decision goes out and does surveys with businesses and end users, consumers, to understand what it is they're looking for with respect to their usage of the Internet.

Marilyn Cade: and you undertook this research as a registry; right?

Ben Turner: no, we undertook this research as the naming and directory services division of Verisign.

Marilyn Cade: okay.

Verisign is a registry.

Ben Turner: actually, no. To be precise, it's a business unit within the division. The division is naming and directory services division, and that's what did the research.

Marilyn Cade: okay. so in any case, I think we're sort of getting close to my ability to narrow down an answer to this.

So you were studying the behavior of end users on the Internet. And I have one more -- consumers and SOHO's, and what they were looking for.

Ben Turner: right.

Marilyn Cade: and that was so that you could develop a service yourself or so that you could provide information to registrars whose job it is to actually serve registration services to end users.

Ben Turner: we actually did it for both. So what we did is we did this research regularly to find services we can offer. We also do research to share information with registrars on what we think will help them sell more product.

Marilyn Cade: okay. I didn't -- and perhaps I just missed it. Has this data already been provided broadly to all of the competitive registrars?

Ben Turner: actually, what we do is when we do research that's relevant to the registrars, we provide it to them.

Marilyn Cade: to all of them.

Ben Turner: to the ones that want it. It's available on our web site, actually.

Marilyn Cade: so all of the competitive registrars would have access to all of the relevant information, which is about the development of services that touch end users; right?

Ben Turner: no, not all of it, because some of it has nothing to do with registrars. The information we provide registrars are things that are relevant to the registration of a domain name.

Verisign runs multiple businesses. We do research to understand what's going on in the market. We take the appropriate research and apply it to the appropriate businesses.

Marilyn Cade: but if I were to ask the competitive registrars, they could go to your web site and find some information, with whatever the caveats are that you're describing.

Ben Turner: right. And I think you would find most of them have attended seminars we've held where we bring in the market research company to walk them through the results.

Marilyn Cade: okay. Is this data presently available to the competitive registrars with the caveats that you've provided?

Anthony Renzette: this is actually used in a number of ways and been alluded to in registry meetings, and we've had the market company come in and give them the data directly, most of the time it's given raw.

Most of the registrars who work with us in that capacity have gotten a monthly newsletter from us, which includes information on renewal rates, things that influence and so forth. All of those things come ultimately from research. So if there's any other questions about how we conduct research, we can do that off-line, I guess.

Steve Crocker: a number of things are queued up here. I'm a little lost. I beg your indulgence but I want to go back to Marilyn's question, basically.

I think she was asking the registry versus registrar distinction, and your answer was this is a business unit of Verisign that's going out and doing this work. But from our perspective, the world is divided up into roots, top-level domains, registries, registrars, and then others, in a sense. And as we all know, one of the things that's at the core of the inquiry and the discussion here is the line that has existed very cleanly for a long time between registries and registrars. And so it's helpful to know from your perspective what actions and decisions and how these things are cast, map onto the registry versus the registrars.

Ben Turner: so we're all clear, SiteFinder service is not a registry service. It does not provide a service of registration of domain names for registrars. So, you know -- and there -- I'm a little -- I'm trying to understand where you're going here.

Steve Crocker: so this is a very core piece, absolutely core piece of Internet technology. A registry has two sides to it, basically. This is just sort of registry 101, if you will.

One side of it is putting new entries in or taking entries out of the registry itself. the other side of it is the lookup process, the serving of queries that come to it. That is sort of absolute, that's why it's there. The numbers you served, what was it? 300 billion lookups during some period of time, that is the heartbeat, the high-speed interloop, if you will, of this operation. Architecturally, there is just no question that when you are making a change to what comes out of that lookup or the way that lookup is dealt with, that's a registry operation. At least I don't know another way to look at it.

Ben Turner: actually the way it's looked at, and it's actually very clear, is you should do that in an RFC compliant way. Which is SiteFinder is completely RFC compliant.

Steve Crocker: so I don't want to skirt away from this. One can have a discussion about whether or not the change is within the RFC --

Ben Turner: the question is are we or are we not compliant within the RFC wildcard?

Steve Crocker: I don't want the change the question. Is there a change to the registry and what actions did you take with respect to that as opposed to what actions did you take with respect to registrars or other external services?

Ben Turner: in the definition of registry operations that you're using, the answer is no, there was no change to the registry operations.

Steve Crocker: how do you reconcile inserting a wildcard into the domain and serving up an alternative response to a query, to a lookup, and say that's not a change to the registry? I mean, that strikes me as not a debatable issue or one that's subject -- it's just a straightforward fact.

Ben Turner: what we have done is we have done an RFC compliant wildcard implementation, which is what we're required to do is to stay compliant with the RFC's.

Steve Crocker: I think there's a useful discussion about the use of wildcards and how they get treated and I acknowledge that.

Ben Turner: that's what I thought today's meeting was about. if we're going to have a discussion about your definition or ICANN's definitions of registry services, we're basically going to have to stop the conversation and turn this over to a different venue. We're not going to have this discussion now.

Chuck Gomes: can I ask a process question, Steve?

Steve Crocker: sure.

Chuck Gomes: I'm totally confused about how what we've been talking about the last few minutes has to do with security and stability.

Steve Crocker: it's crystal clear. If you make a change to the registry, what is the impact of that, what are the ripple effects?

How does that change the interaction?

Chuck Gomes: we've had no changes to domain registration services.

Steve Crocker: I don't know what domain registration services is intended to convey, but is there change to the registry operations and the answer is unequivocally yes.

The questions that follow from that is does that matter, is that an important change, how was it done, a whole series of other questions, acknowledge all those questions. But there can be no question that was there a change made to the core registry service.

Ben Turner: there was an RFC compliant change made to the registry services which is well within the purview of the registries.

Steve Crocker: so let me translate that. You said yes, we made the change but we believe we made the change within the allowed range that the protocol definition -- the RFC specifies. Is that a fair reading?

Kevin Golden: hi, Steve. My name is Kevin Golden. I'm counsel to Verisign and I think the difficulty we're having here is certain terminology is being put out there that is loaded with legal significance.

And so there's a reluctance to go down that path in what should be a technical discussion.

So we're happy to have a discussion on the legal nuances of some of this terminology with ICANN's counsel, and have done so on many occasions, but I don't know that it's productive to continue going down this path because it's loaded terminology, and I think that's what you're seeing.

Steve Crocker: I see. thank you.

All right. So with that I think it's fair to move on to other kinds of things. Jim, you've been monitoring some things coming in from secsac-comment? And I'm sorry, are there people queued up for the floor? Let me do that first.

David Lesher: good afternoon, David Lesher with NRK Research.

I'm looking at the page labeled issues analysis but unfortunately does not have a number but it was reasonably early, and it says issues more likely to occur with at least moderate impact and how addressed. And I count, one, the first item and the last item as being addressable by Verisign. And I count one, two, three, four, five in the other hand that have to be addressed basically by the rest of the user community.

My question is what's the estimate as to the cost impact and the time to develop these answers and deploy them.

Ben Turner: what we looked at here is how we mitigate.

David Lesher: not the cost of mitigating.

Ben Turner: we did not go into the detail.

David Lesher: next question on the TRP work product, there's another question that says some delays, some things back and forth but in the long run just sort of say not important. Has there been any analysis as to the total additional bandwidth or total additional time this will take on a worldwide basis?

Scott Hollenbeck: from the TRP's perspective, no.

What you see here is the total what have was discussed by the TRP based on their analysis of impact.

David Lesher: is it fair to say in the words of the current administration this is a cost shifting, from Verisign to the community?

Ben Turner: I think that would be unsubstantiated unless you --

David Lesher: I'm addressing the things stated here.

Ben Turner: these are the issues we looked at and what we looked at, Scott went through.

David Lesher: but there was no or very little detail as to the work actually required to solve the problems this creates. It just says there's a minor problem. Well, minor depends on how big your sample set is. It's a big sample z that's my only point. Thank you.

Steve Crocker: question from the committee here, both Steve and Paul are queued up. Let me take Steve first.

Steven Bellovin: I've got several questions. One, of the three anonymous reviewers, were any of them from the IETF leadership, work group chairs and so on other than the Verisign people?

Scott Hollenbeck: no. When we asked people with those sorts of affiliations if they're interested in participating, their answer was no.

Steve Bellovin: okay. it's too bad since that's where the center of expertise in protocol analysis.

Scott Hollenbeck: we made an attempt to reach out to those people and a few of them, because I think of the controversy of this, said we don't want to be involved at that level.

Steve Bellovin: I'm not blaming you. I'm saying it's unfortunate.

Question of software updates, there have been a number of surveys in the last three years or so on how long it takes people to make upgrades even when there are critical security problems. and a remarkably high percentage of people never upgrade their machines. And even for security things, even for people who should know better, such as SSL, SSH, servers all show the same patterns. These are things run by more sophisticated people than ordinary consumers. I heard a number, I think it was 40% to 60% of the users never even clean up their infected home PCs when they're infected. So do you actually expect people are going to make any of these software changes or simply wait to buy a new computer with whatever Microsoft ships in years from now?

Scott Hollenbeck: the discussion the panel had around this would probably answer this no for the reasons you just described. There are lots of people out there running old operating systems for various reasons.

From the panel's perspective, they were trying to discuss what the mitigation strategy would be, not the likelihood of the mitigation strategy being viable. Does that answer your question?

Steve Bellovin: it turns to a previous question about cost shifting but yes, it answers my particular question.

Chuck Gomes: let me add one more thing. Chuck Gomes again. There were a total of four companies that actually contacted us when we made information available to do that, and identified specific problems and we assisted them with those.

If there are companies or individuals, for that matter, that have issues that they can't resolve, we do have an e-mail address that's been made publicly available where they can contact us, and we will help them with it. But what you're hearing, we need -- what you're saying is all good and fine, but we need the evidence of it to be able to work with it.

Steve Crocker: did you write RFC's that lay out the mitigation strategies?

Chuck Gomes: no.

Steve Crocker: thank you.

Did you have more, Steve?

Steve Bellovin: I think that's it for now.

Steve Crocker: okay.

Scott Hollenbeck: Steve, if I could answer your question partially, no, we don't have an RFC out, of course if we did it would take far longer than a few months to get it published. We do have a guidelines document that we wrote and released a few months ago. This is a document that recognizes that registries are doing this. There are issues associated with wildcard, and we are looking at a document, and using the formal processes using the IETF processes.

Steve Crocker: I'm glad you're looking at it. I commend that process to you.

Thanks.

Paul.

Paul Vixie: yeah, first I have a question for Matt and then I have one for Ben Turner.

So Matt, in your slides you said that one of the things you were considering for the future of mail related to the SiteFinder and wildcard was to put in an MX record pointing to a nonexistent name.

Thank you for finding the slide. That particular name would be found because a mailer, which didn't get the glue, would then query for an A record by that name and they would get the SiteFinder name.

Matt Larson: not unless it were delegated to a zone who's a apex had a (inaudible) domain.

We could do it in an undelegated TLD.

Paul Vixie: I'm not suggesting you do it at all. I just wanted to clarify that point, since there is a wildcard -- or would be in this scenario, a wildcard under dot com, you would not be able to find a name that doesn't exist so I thought your slide was inaccurate in that minor technical detail.

Matt Larson: you're right, I did not say this would have to be delegated, and it would be a -- it would be a negative response but not strictly speaking a nonexistent domain response.

Paul Vixie: that would be an empty response; strictly speaking, a nonnegative response. but that's okay.

Ben, I have a question. Actually, I don't know who at Verisign to address this to, but your slides came closest to this point. You surveyed some number of web browsing end users in order to find out what they thought the impact was on their web browsing experience; however, other than by accident, that population is not your customers. And what I'm now curious about, you have two hats. As a registry, your customers would be other registrars. As a registrar, your customers would be other registrants. Did you survey those two populations to find out what the impact was on them; in particular, the registrant population is the one that's affected because they are the ones that are -- whose names are being mistyped.

Ben Turner: yes, I don't have this -- I don't have the specifics in our end-user surveys that we hit registrants. The probability is we did hit registrants.

Paul Vixie: so you didn't specifically survey any of your own customers but, rather, just essentially third-party end users who happened to be using web browsers were the targets of your survey?

Chuck Gomes: I'd like to make something very emphatically clear. Verisign naming and directory services is the home of the Verisign com/net registry with which I'm associated. The Networks Solutions registrar is not associated with our division at all.

Paul Vixie: Chuck, thank you for that but I want to ask a follow-up question. It's still a wholly-owned subsidiary?

Chuck Gomes: that is correct.

Ben Turner: what I'd like to understand, though, is what does this have to do with security and stability and what does this have to do with wildcards?

Paul Vixie: I just wanted to know if you had tried to survey any of your own customers, and it sounds like other than by accident, you didn't. That's all I wanted to know. Thanks.

Ray Plzak: as a follow-up, I would like to ask you the same question about your survey of the web users.

Ben Turner: and that question would be what?

Ray Plzak: the one you just posed.

Ben Turner: I'm confused.

Steve Crocker: it's probably helpful to restate it.

Ray Plzak: his question is what did the things that Paul have to do about security and stability in wildcards?

Ben Turner: my question was with respect to there's a subsidiary of Verisign called Network Solutions and there's a division within Verisign called Naming and Directory Services which does not encompass Network Solutions. What does that have to do with this discussion?

Ray Plzak: thank you for the clarification. I'll ask something else later.

Paul Vixie: Steve, I'll follow-up on that. I wanted to get clarification on the survey that had been done. Just wanted to make sure I understand what had been presented.

Steve Crocker: so go ahead.

Paul Vixie: I have what I need.

Steve Crocker: thank you.

Let me poll people on the phone. is there anybody who has been listening who wants to ask a question?

Ram Mohan: Steve, this is Ram. I have a couple of clarifications regarding the survey.

Steve Crocker: go ahead.

Ram Mohan: the first question is, the survey shows a comparison of the SiteFinder, the web page. The first question is was the SiteFinder comparison made in comparison to MSN search, which is a default in Internet explorer, or was it in comparison to a 404 error page, the first question.

The second question is did the two survey vendors that were used, did they use the same methodology? I thought I heard that the same methodology was used and later I thought I heard maybe a different methodology was used. A follow-up to that, are the results that are put out a blended result of the two surveys or the different results from the two surveys?

Ben Turner: your first question was already answered.

Your second question was did we include -- was it -- I'm confused on the second one. Was it blended data? Is that what you're asking?

Ram Mohan: yeah. You used two vendors, and I thought you had said earlier that the same methodology was used, and the results -- I'm unsure or unclear as to whether the results that are presented, whether that's across the various surveys done by different vendors or is it individual, independent results?

Ben Turner: it would be invalid if we blended the results so we did not blend the results.

So we used two different research companies. One did a U.S. sample.

That same one did a U.K., Germany and China sample. That's the information that's displayed in the charts. the other, customer strategies, did another U.S. sample. We will asked them to gather testimonials, and that's on the testimonial page.

Steve Crocker: let me ask a question within that, a question from Thomas Roessler. What portion of those surveyed in Germany and China did not speak English?

Ben Turner: I don't know the answer to that off the top of my head.

Steve Crocker: all right.

More questions from the floor.

Rick Wesson: Steve, this is Rick Wesson and I have a question.

Steve Crocker: Rick?

Rick Wesson: yes. and this is, again, for the survey. My question is why does a survey of users show impact on security or instability versus a survey of applications? If Verisign could just discuss why they chose to survey users versus applications.

Steve Crocker: the question is on the screen there.

I think we asked that a little bit earlier as to whether the survey had focused on security and stability issues. But you may want to add to that.

Rick Wesson: the question is specifically why they chose to survey users versus applications and why stability or security affects users and not applications. I take it there's some philosophy that Verisign has on this.

Ben Turner: we actually did survey application users in our launch testing, and then we also made available other ways to get information to us regarding impacts on applications post launch.

Rick Wesson: if you surveyed the effect on applications, where is the raw data? Or can you make the raw data available?

Ben Turner: yes. So the current slide summarizes the application testing we did. So if you look up here, you can see that we went across 13 categories, 37 protocols, and 53 different implementations of protocols.

Steve Crocker: every time I see that, I'm inclined to ask, did you engage the open community, the IETF and operators and IAB and so forth. And I think the answer, as fast as I'm inclined to ask that, I guess your answer is you view this as a proprietary process.

Ben Turner: in that we did, you know, open it up to -- we went out to 600 companies, asked them if they wanted to participate. We had 55 that were interested in participating. Out of that 55, 35 actually wanted to test.

Steve Crocker: I'm struggling here, but the thought that is -- that wants to get out is that the Internet was really built not by a set of companies that built a closed, proprietary system but by a much more open process than that with, you know, arbitrary numbers of individuals coming from every quarter and dealing themselves in as they want rather than being selected.

Rick Wesson: Steve, would you please ask the Verisign people to make available that protocol survey?

Steve Crocker: I think you just did.

Rick Wesson: thank you, sir.

Steve Crocker: so that's the question. Would you make a protocol survey available?

Ben Turner: sure. What we've made available here are the results. The survey itself is proprietary and just like other companies, we're not comfortable making that available in an open forum.

Rick Wesson: just to clarify, so the data would not be available for outside evaluation; is that correct? The raw data.

Ben Turner: I think throughout this process, we've been willing to sit down and discuss the data we've collected. That's what we're trying to do here. If you have specific questions, it would be helpful if we could know them and we could try to answer them for you.

Rick Wesson: the specific question is can I have the raw data? Or can the raw data be made public?

Ben Turner: I think I just answered that question, actually. Really, if you have any specific question --

Rick Wesson: (inaudible) I can't pick it up on the web thing, so if you could just comment; I'm sorry, I didn't catch that.

Steve Crocker: Rick, I realize California is different from Washington, so let me use the skills learned in Washington to translate what you've heard here, and the answer is no.

(laughter).

Rick Wesson: thank you.

Steve Crocker: Olafur.

Olafur Gudmundsson: hi, I'm Olafur Gudmundsson. First question to Ben or comment, you're absolutely correct, adding wildcard to (inaudible) is fully RFC compliant. No question about it.

The issue is for 13, 15 years, the com had been without wildcards, and then one night it appears without a warning. Is there -- I hate to use this word I'm going to use, because I think your lawyer is going to come back and hit me in the head. Does this non-use of wildcards for 13 years mean that there was an unpublished service-level agreement with application writers and users of the com zone that this would not change overnight?

Ben Turner: no, we look to standards as the guidance.

Olafur Gudmundsson: standards are not the full answer to everything.

There is social behavior also involved, there is courtesy, there is a kindness to others.

I've been trying many years to have wildcards thrown out of the DNS standards. Every time people tell me not to do it but every time people start using them there are more problems that come up. So yeah, I think the biggest mistake you guys made, and it's really biting you, is the total lack of communications up front. Because the rest of the top-level domains are irrelevant compared to com and net.

Ben Turner: we agree that one of the lessons learned out of this is it's important to go out and do pre-notification on a service like this.

Olafur Gudmundsson: yeah.

I'm going to repeat a comment that I made to somebody at Verisign when they asked me about this a few months beforehand. I worked prior at one of your wanna be competitors, it took me about 20 minutes to kill the project, because a spelling correction is totally hopeless because you don't know what language I'm going to speak. So (speaking in foreign language).

Mark Kosters: Olafur, to be fair, that particular conversation didn't hold up because if I recall, your previous employer did put up a test after --

Olafur Gudmundsson: after they laid me off.

Steve Crocker: you'll help the transcribers transcribe from Icelandic into --

Olafur Gudmundsson: yes, I said I don't understand error messages in English. I said that in Icelandic.

Steve Crocker: they are fabulous but I suspect that tested even their skills.

Olafur Gudmundsson: and Scott and Matt, Internet draft deadlines is either today --

Scott Hollenbeck: Monday.

Olafur Gudmundsson: monday? So get it out so people can look at it, without the note that this is Verisign proprietary information like the current document has, which causes some people not to look at it.

There is already one Internet draft come out that where people are proposing to change the protocol to describe synthesized answers.

Scott Hollenbeck: the guidelines document has no such proprietary markings on it. An earlier version of the implementation document had those on it. We have no intentions of taking that out into a public forum and submitting that to the IETF process.

Olafur Gudmundsson: please do, because it's a good point, and registries who are not willing to do their homework like you guys did, we don't want them to make mistakes.

Steve Crocker: to your point about deadlines for drafts, this is a submission that happens just before IETF meetings and then the window opens up for unfettered input until a similar window several months later.

Olafur Gudmundsson: yes.

Steve Crocker: so I don't know how soon it opens up after an IETF meeting.

Olafur Gudmundsson: middle of November.

Steve Crocker: so that's right after the IETF meeting, basically.

Olafur Gudmundsson: yes.

Steve Crocker: that's just a clarification.

David Conrad: this is David Conrad. I want to clarify at least my view of some of this in that standards compliance is somewhat of a red herring in this discussion. Wildcards are a part of the standard, but it is perfectly standards compliant to create a server that doesn't do anything; right? Saying something is standards compliant is sort of a cop-out from actually trying to interoperate with the technologies that are available on the Internet.

The working group that Olafur is involved in, the DNS XT Workgroup, is not the right place where discussions about the operations and the characteristics of the wildcard implementation within TLD registries.

The appropriate workgroup for that would be DNS ops.

Steve Crocker: who chairs that group?

David Conrad: I don't remember.

Ray Plzak: it's Rob Austin.

Olafur Gudmundsson: I talked to Rob before I came here.

Steve Crocker: does that mean you're speaking for him or he's speaking through you?

Olafur Gudmundsson: he agreed to having this document in an open forum as a discussion point would be a very useful thing.

Steve Crocker: good.

Sir.

Tim Ruiz: Tim Ruiz from Go Daddy Software.

Just a discussion earlier on this, that's regard to SiteFinder's home as an entity, that's with the Naming and Directory services, not necessarily or not the com/net registry business unit. And the com/net registry business unit's only role with SiteFinder is it made the wildcard entries that made it possible.

Ben Turner: that's correct, it's not part of the com/net industry.

What the com/net registry does is the registration of a domain name and the resolution of a domain name.

Steve Crocker: Steve.

Steve Bellovin: I just want to turn back to the question of software -- I'll sit back a little bit. I want to turn back to the question of software updates and timing and so on.

A number of the recommendations spoke about protocol changes to indicate that it's a synthesized response. That at a minimum it was going to take several months through the IETF process. How long do you estimate it will take developers of relevant pieces of software to develop their changes, test them, to test new versions of the software with lord knows what other changes in their environment? In other words, how long do you think it's going to take before these changes can be safely deployed by lots of people?

Scott Hollenbeck: Steve, you probably know better than I do what it would take for IETF standards to be deployed. Typically, that time frame can be measured in months or years.

Steve Bellovin: it will take months at a minimum to get the standard through the IETF for the protocol change. But I was actually referring to the software engineering effort to get it developed and tested, and then the deployment engineering. You guys run very reliable production servers, and I know from my experience you don't do that by upgrading to the latest version of every patch that comes out from your operating system vendor.

Scott Hollenbeck: quite right.

Steve Bellovin: lots of other people have similar concerns, especially about things like mailers, for example. So I'm asking what your estimate is how long a site should test a new version of a mailer before they're willing to cut over to it, one that would have this protocol change incorporated.

Scott Hollenbeck: our standards to deploy a service like that would, typically, as you said, first off initial versions don't get deployed. You let somebody else plow the ground and get the bloody nose. And after that there's still a testing effort measured in weeks or months before we're ready to deploy any kind of new software.

I assume other companies who operate critical services would have time scales measured in the same way.

Steve Bellovin: I agree. I know what -- I know how long we beat up network elements before we put them into our network, and it's an interactive process with the vendor after getting things fixed up in our test lab.

And I'm trying to suggest that trying to get a handle on numbers like this are important if it is decided to go reinstate the SiteFinder service, the timing of the notice that's necessary has got to factor in the delays to let people upgrade and test software, and that's even after everything is defined, which has not happened yet. So I'm asking -- as you say, you test for weeks or months and other people who use e-mail that's mission critical would as well and that's one of the things that would need changes per your own survey.

Ray Plzak: Steve, just a process clarification, my understanding is the way a name gets into the com or net zone, a registrant applies to a registrar and then it goes and gets through process and gets put into the zone; is that correct?

Scott Hollenbeck: not necessarily.

And I say not necessarily because, I mean, at the high level you have the process correct. we receive automated requests from registrars and then names are in our database before they're published to zones.

And there are a number of factors that might prevent a name from being pushed to the zone, such as for example if there is no delegation information, no name servers, it doesn't go in the zone. If the registrar or the registry has the name on an internal status we call hold, it doesn't go into the zone.

At the high level, though, you're absolutely right. The registry gives registrar information, registry publishes information in the zone file.

Ray Plzak: and this originally starts with a registrant of some kind.

Scott Hollenbeck: typically. Registrars register names for themselves where they play the role of registrant, yes.

Ray Plzak: what was the process followed to put the wildcard names into the zones?

Ben Turner: there are no wildcard domain names in the zone.

There's a wildcard entry in the zone.

Ray Plzak: what was the process to do that?

Who made the decision?

And what would have been done to notify people this had been done?

Mark Kosters: I'm having a hard time seeing -- who asked who to pull the plug or insert the plug. I don't see how that's relevant.

Ray Plzak: what's relevant is you're changing domain. Is the domain administrator responsible for it?

Mark Kosters: all the names are registered, there's no change whatsoever, right?

Rick Wesson: mark, I have to disagree with you, this is actually very relevant and interesting question. This change affected vast majority of the Internet. And how many applications consider availability and so it is interesting to ask this question as to what entity put this record into the zone?

I think that's what Ray was asking there.

Steve Crocker: so, yeah, let me try to make this very -- as simple and if it's not if you guys don't agree then we come back again. I think there are two things.

The entity that made changes was the entity that was operating the registry. And as opposed to any registrar and I think that is pretty straight forward. The next question is, does that affect security and stability and that's exactly what we're here discussing, there's clearly enough input that triggered our activity that indicates that enough things changed so that all by itself just the fact that there was a significant change is important from a stability point of view. And then there's further questions as to whether or not there is a stable resting point that one can reach after all of this and it's appropriate or not and that's sort of more complicated question. But to your question of who made the change, the answer's the operator of the registry, full stop.

Mark Kosters: I think we have a valuable set of data for a 19 days of service was operational. And it would be very interesting for people to actually look at what things have changed and actually present hard data to this committee to say, okay, how did it affect as opposed to seeing things painted in a -- perhaps a subjective manner.

So, I think that going through some of these things in terms of who did what, when where doesn't make a lot of since, we have a love the data to cull from and let's look at this and go forward.

Steve Crocker: I think in fairness if you want to say, well, one might object to the way it was done but nonetheless we have data that we should look at. One consequence that also sets a precedence of running those kinds of experiments. That is not only did the system change how did the processes for making changes change, if you will.

That is does this now encourage or legitimatize or make changes to the system by somebody else running an experiment like this, there are other actors in the system, I mean could make changes and then say, well, we did all of this analysis ourself and public process are a little slow and bulky, so we've decided to move things along more smoothly here a little faster.

One could say Cisco runs most of the router software and decide to usefully improve the operation by issuing upgrade to its software, for example. Or one can pick other large players.

Rusty Lewis: this is Rusty Lewis, if, I could make a comment, Steve to, your point about the data that the committee has amassed at this point in time just to follow up on Mark Kosters' remarks.

That's one of the things we're here today to try to do, we're trying to exchange the data that that we have and get data back from the SECSAC and broader community. In all honesty we've heard a lot of anecdotal data we're trying to get our arms around that data to put that data in some sort of context. As Scott Hollenbeck said the results that we got back from the technical review board were that while there was some impact it was inconsequential. And the service calls that we've handled has been a handful of companies. To the extent you have -- extent that you have hard data that there is substantial or there is even a security and stability risk here, we want to hear that. But we don't want to just presuppose that this service that end users we believe clearly want and needs to be forever buried. That doesn't make sense to us. So we've reached out here today and presented an awful lot of data that we've amassed before we launched the service and after we've launched the service. We have continued to reach out to the SECSAC and asked for the hard quantifiable evidence that you have to refute our position that there are a handful of companies, and as we've said we have customer service reps out there, Chuck Gomes pointed out the fact that we are willing to work with companies, to address these issues. but in the grand scheme of things the supposed stability and security issues that you're alluding to, we just don't see the evidence. So we're here, this is supposed to be a two-way dialogue. we're looking for data from you and we're sharing our data as well.

Steve Crocker: actually it's supposed to be a discussion and query for us to understand what's going on and then speak to ICANN and to the community. The actual flow of information as it should be is for -- when there's a change of this magnitude that it be vetted through sensible process that are designed to assure stability, not that to act as a market research service for you, but why don't we just hold that and just move on to the next set of things, if we can.

Rusty Lewis: if I might just make a follow-up comment. Fair you have, it's your process.

But you've asked us to come and share the data and we think that we have demonstrated all of our efforts before the launch and post launch that we've bent over backwards to not adversely impact the stability or security of the Internet. We do believe that the data that we presented does overwhelmingly support that conclusion to the extent that's not the charter of your committee we fully respect that, that's your right to decide how you want to report your findings to ICANN.

Steve Crocker: well, I appreciate that. And let me emphasize that we view the inputs as having two completely separate parts. One is the input of the data the other is the assertions and packaging that you have on that which is perfectly within your purview.

We are necessarily restrained in responding in kind to that until we've had time to deliberate over that. And we will do so in fullness of time.

Are there other questions from phones, from the floor? ah-ha, people.

Jim who probably asking on behalf of somebody from the net.

Jim Galvin: I have three questions from SECSAC comment.

Steve Crocker: I see you've got two cords that are almost stretched to the limit there.

Jim Galvin: that's okay. One is the -- he's going to fix them for me.

Somebody asked, as follow-up question that Verisign did we correctly hear them say that they're not collecting any personal data of course and they said that multiple times that's a clear statement. However can you comment on the presence of the web bug in the SiteFinder webpage?

Scott Hollenbeck: the web bug exists. That was asked at our last session of we have plans to cut back on the information that's being passed from via -- the web bug to the URL. We have one of our development managers, Joel Nylund, if you wanted to say anything more about that.

Joel Nylund: other than we're passing the whole URL we plan to (inaudible).

Scott Hollenbeck: he said what I said. it's going to be changed to pass back only the minimal information.

Steve Crocker: is there an opt-out mechanism?

Ben Turner: the way we do the web bug is compliant with the standards that exist. It is a typical implementation for this type of bug.

Steve Crocker: I'm speechless.

Jim Galvin: moving on to question two.

And you asked the questions, Steve, you commented about opting out.

There is one distinction between Verisign commented about combining comparing SiteFinder to MSN and AOL. and those kinds of search pages but there's one characteristic difference that you can opt out of those and you really can't out of the SiteFinder service, Verisign is uniquely positioned to own that consumer base and that particular service. Do you acknowledge that discrepancy or do you have a comment about that? We're trying very hard to avoid using that word I'm sure you know.

Rick Wesson: which word?

Jim Galvin: not to say monopoly. That's in the question here I was trying to rephrase the question to not use that word.

Ben Turner: so, as was illustrated after the launch of SiteFinder there were a number of ISP's that decided not to allow SiteFinder responses and implemented ways to avoid the SiteFinder response. We're supportive of that.

Jim Galvin: the distinction is for users, though. I mean clearly technically savvy person or service provider could certainly provide these things on behalf of its clients and customers, but users can't opt out of the service.

Ben Turner: we are also exploring ways that a user could opt out of the service.

Jim Galvin: one other clarifying question. Did we correctly hear Verisign say that they had changed the gTLD service for dot net and dot com several times before they dot com and by enabling the wildcards.

Ben Turner: we did do live tests before that. So we could understand the behavior.

Jim Galvin: one follow-up question, why did they not inform the user community of this?

Ben Turner: that we were testing?

Jim Galvin: I presume, yes. that you were affecting the user behavior. Ben Turner: we -- in the test phase we designed the tests so they lasted for minutes then turned it back off so we could understand the data.

Jim Galvin: okay. that's the three questions from comment.

Steve Crocker: thank you.

Ben.

Paul Vixie: there are some questions coming from the phone.

Steve Crocker: let's take Ben first go back to the phones.

Ben Edelman: Ben Edelman from the Berkman Center, all of the talks of surveys that is something of distraction from security and stability, but putting that aside since I have a degree in statistics I thought I'd just jump up and say what they taught me in class.

About publishing your surveys. What they teach in class is you've got to publish your survey, you've got to show anyone who is supposed to interpret your results what questions you asked because otherwise they wouldn't be able to assess whether the questions were biased. There are large number of ways in which questions are known to have affects on answers, for example, recipients of surveys just love to answer questions in the affirmative. Do you prefer x to y? The people are more likely to say, yes, they prefer x than if you say, do you prefer y to x then they say they prefer y. Whatever you say in the positive sense is likely to get a greater response. There are similar requirements not just in classes and sort of a Boston kind of a mode but in a D.C. framework that federal rules as to admissibility of surveys as expert reports in court. Have the same requirements I guess based on that reasoning generally. So, inasmuch as the committee has some concern as to the lack of availability of the survey, that might be helpful in assessing why it rubs us all the wrong way.

Steve Crocker: thank you. I think that was at the heart of the questions that we started out with about the availability of the survey of the structure and intent of it and so forth.

Questions from phones?

Paul, you were suggesting that people were asking on the phones? who on the phone has a question?

Doug Barton: I have a couple of questions.

I'm getting a little bit of feedback here, is that true in the room as well?

Steve Crocker: just go ahead.

I'll try and figure it out.

Doug Barton: one of the Verisign representatives mentioned a SpamAssassin and as a representative of SpamAssassin, unfortunately I don't think that's exactly accurate in return -- in reference to the issue of unresolvable domains because the Spam that is in their corpus is heavily biased towards domains that actually resolve given the fact that most mailers already filter out stuff that doesn't resolve before it ever gets to SpamAssassin.

So that's a little bit of a red herring there.

Second, one of the Verisign representatives mentioned a warning period before re-enabling the service, I'm wondering if we can get somebody to commit to a period of time and commit to actually giving a warning to the community before the thing is re-enabled.

Ben Turner: so we are committed if we decide and when we decide to move forward to giving notice that we're going to do so. We're very open to getting feedback from the community on the best way to do that.

Doug Barton: and what kind of time period are you guys willing to commit to at this point?

Ben Turner: we're open to suggestions.

Steve Crocker: you have a question?

K.C. Claffy: I'm struggling with the slide set for details on (inaudible) design stuff. I understand Verisign keeps asking the community for hard data, I'm sorry that they don't have any to offer at the moment I think this is outrageous to expect the community to be able to turn on a dime here and come up with legitimate sound methodological approach. The organization that is chartered to evaluation this kind of thing the ITS. It still boggles by mind why this sort of change wouldn't go through the IETF.

One I would like that to get back on the table if it's been on the table at all I understand Verisign's reasons for not going to the IETF I don't think they're good enough quite frankly. Two, I'd like more detail on slides eight, nine, ten, you do talk about some of the data but you do things like margin of error, classified percent at 95% confidence level I'd like to know how you got that. You say external statistician using certified sample methodology and analysis that doesn't mean -- doesn't make me feel any better. Although I note that if you're outsourcing this analysis anyway why don't you outsource it to the IETF. And again it's going to take time to do an analysis of this of all the infrastructural facts, going to take people sitting in a room together and talking about the possible implications and tests designed all this sorted of stuff that the IETF is really quite good at.

Steve Crocker: I know she was a little hard to hear we were able to read the -- we were able to read the question off the screen.

Ben Turner: I think she suggesting that we should outsource our research to the IETF is that --

K.C. Claffy: not exactly. I'm definitely suggesting that this process be evaluated to the existing community organization, is that were designed to evaluate this kind of process and change. I have not seen a compelling argument for why not to put this through the IETF or not to work with the community on this.

And second part of my question is, or my comment is, you're asking for a hell of a lot of hard core science and engineering from the community with no notice and offering no resources to do it. So, it's not going to happen in the time wise you need it or want it. And then the third part is, if you're going to ask for the community at least you can give us the same, which is real detail including test leads and maybe scripts, the kind of thing you'd have to do to prove that this were a good -- not going to affect the infrastructure through a scientific leadership.

Steve Crocker: do you want to attempt to response?

Ben Turner: not sure what the question is, actually. I think it's a statement.

K.C. Claffy: more detail on your experimental design.

Slides 8, 9 and 10, at least of your 49 slides.

Steve Crocker: I think there are sort of two parts if I might to what K.C. is getting at. One is that the technical level of getting the data out there and getting it looked at, the other is that the process level for discussing, for conducting these kinds of investigations and for gathering this kind of data and whether the process that has been engaged in of your activities are consistent with the normal modes of behavior and collaboration that have existed previously in the Internet, particularly when changes are being made at the core.

You may view what my interpretation of her question is, yet another speech and if so that's fine. we can just let it sit there.

K.C. Claffy: thank you, Steve.

Rick Wesson: I have a question.

Steve Crocker: hold off for a minute, rick.

Ben Turner: so, what Verisign did just like we do with all of our products we went through a pretty robust product development process that is a standard way to do product development and testing.

We do look to work with the community, we did reach out to 600 companies of which we tested directly with 35, 55 after going through further investigation decide they didn't need to test.

Steve Crocker: thank you.

I'm watching the clock a little bit, we're well into where we intended to take a break. And we have more to cover today.

Rick, you were queued up, is anybody else, do you want to proceed or have we basically covered the ground?

Rick Wesson: I have one last question.

Steve Crocker: fire away.

Rick Wesson: has Verisign any intention to do undisclosed testing on non-delegation records in the com/net zones, any further testing?

Steve Crocker: the question relates to testing on non-delegation records?

Rick Wesson: yes. I'm just asking if Verisign is going to do further testing without disclosing that they're doing testing?

On non-delegation records.

Steve Crocker: I see.

Ben Turner: I'm just reading.

Steve Crocker: if I understand it I think the question that Rick is asking has to do with doing the kind of testing that you related before where you did testing in advance for short periods of time without announcing it and that -- I think Rick is asking if you plan to do more that have kind of testing.

Ben Turner: if we decide to move forward with the service we'll do the -- what appropriate testing is required to ensure that we offer a service that is stable and secure.

Steve Crocker: okay. good.

Rick Wesson: not actually good, Steve.

What I'm concerned about is non-disclosed testing on -- in any kind of non-delegation data that the community doesn't know about, no one knows that this is going on. But absolutely has to do with the stability of the zone.

Steve Crocker: I didn't mean good that it was good thing to do I just meant good that now we understood the situation.

All right.

Let me choose this time to bring this section of the meeting to a close.

Everybody take a stretch, it is my goodness 3:45, almost exactly. let's pick this up at 4:00 sharp. we'll begin with Ben Edelman and thank you very much.

John: for the people on the phone line we're going to keep the phone active so you can just hang on to the call if you want and we'll be back shortly and we'll announce it before it starts.

(break)

Steve Crocker: welcome back. We have Benjamin Edelman from the Berkman center for the Internet Society, Harvard Law School, talking about some measurement results, some real hard data. Take it away.

Benjamin Edelman: I apologize for not being here last week when I suppose this would have fit in more neatly but I was obliged to be on an airplane at that time.

Just a little bit as to the motivation as to why this is worth measuring. I don't suppose this group needs a hard sell as to why data is helpful and useful so I'll spare some of that in the interest of catching up on the schedule, but I do want to do the second, third and fourth bullet points, what it would mean if we found that SiteFinder had been disabled by a lot of networks, what it would be mean if we found that SiteFinder hadn't been disabled much of anywhere. This is ultimately something for a lot of people to debate, and I don't mean to presuppose that any particular arguments are winners here. But since some of it can be a little bit counter intuitive, I'll report the two lines of reasoning that have seemed most obvious in talking to other students, professors, researchers, technologists about it. If we were to find SiteFinder was disabled in many places, I think that would tell us two things: one, that a lot of people didn't like it and wanted it to stop, but two, paradoxically that it wouldn't be so bad if it started again because it wouldn't be working anywhere because everyone is already blocking it. I don't mean to endorse that particularly, but it's a line of reasoning to bear in mind when looking at the data. And the flip, if we found that SiteFinder was disabled in only a few places, would that tell us that no one minded SiteFinder and it was actually just fine with the ISPs, or would it tell us it was hard and costly and a big pain to disable SiteFinder. And so it's all the more important that SiteFinder not be turned back on. I don't mean to endorse any of that, to be sure, but it's worth at least thinking about in attempting to interpret these results. So in trying to figure out which networks have taken steps to disable SiteFinder, I want to suggest three distinct methodologies that could be used and then I'll put some detail into the third methodology I've actually attempted to date. The first is surveys, form submissions, putting up generally a web page and asking network operators to come to my web site or ICANN's and tell us what they're doing, maybe tell us why they're doing it, and then we'd get at least some anecdotal data. There are some obvious problems in proceeding that way. There's a burden on respondents.

Every time you ask NANOG readers to do something, that's taking three minutes of time of a thousand different people or 5,000 different people, not to be taken lightly. There's a major representativeness problem. Would the people who took the time to take the survey be representative of registry operators? Probably not. You'd get a lot of people who didn't like SiteFinder who took the time to fill out the form, a bunch of people who really liked it, and the masses in the middle would be represented, they didn't care one way or another so they wouldn't answer at all. So there would be difficulty interpreting the results. Finally a concern as to accuracy. If you get someone who purports to speak for AOL but doesn't have an address at corporate.aol.com, do you believe they're telling you the truth because they tell you so? It would be nice to verify what they're saying in some additional way beyond merely accepting their form submission. But all of that hasn't stopped me so much that I didn't give it a try. I put up the small web page shown in the shot in the upper right corner and got some responses to date, an official response from AOL saying they did block SiteFinder. A response from Earthlink, someone who seemed to speak for Time Warner Cable but maybe didn't. This is not so clear.

This is the kind of accuracy concern that's latent in the method. And then any number of smaller organizations, San Bernardino County School System, certainly one of several dozen at this point that have submitted their responses. I'll publish the list. I suppose I can publish it in real time so you can see what people submitted as soon as they submitted it, now I've got enough submissions, 20 to 30, that it might be of some value. I'll go ahead and do that as soon as I get back to the office tomorrow. If we get Spammed by sort of a Jeff Williams character, we'll deal with it as it comes but at least publishing the full data and survey as well as submissions and responses will let people take the data for what it's worth. Second, looking at the SiteFinder log files.

This is a particularly appealing approach in that it would provide a complete view of everyone looking at SiteFinder. You'd see what web browsers made requests for sitefinder.verisign.com, what IP addresses the people were coming from what web browser types they were using, you would really figure out quite a bit about what was going on. And we know Verisign has this idea, thanks to the tracking cookie. They're demonstrating an interest through the web bug and otherwise in getting good information about what sorts of users are visiting the site and how they are using it. To the best of my knowledge, this data is only available so far in summary statistics, the sorts of things that come out in press releases and comments from Verisign's public relations office. It would certainly be interesting if more of the data were available in a way that was meaningful to researchers for conducting their own analysis. With that in mind, I wrote to Tom Galvin, a spokesperson for Verisign, who said actually he couldn't give me the log files. He says there's an operational security concern as to the log files. I'm not completely clear on what that might be, but maybe it's important that I not even know. Certainly, there would be a potential privacy concern as to a bunch of IP addresses of users accessing the site. On the other hand, maybe you could obfuscate the IP addresses in some helpful way, taking away the low order byte of data and then there wouldn't really be any particular privacy concern. These are the kinds of things that other providers of data do when wanting to make that data available to researchers or others. Certainly it seems like Verisign could do that here if they wanted to facilitate this method of analysis.

But so far, I can't do it. Maybe Verisign can do it; to some extent, already has. That's reflected in their public comments to date.

Finally, it's possible to draw some inferences from data that is available, either publicly on the web or via special arrangements for researchers. The idea is to get data about selected users or selected networks' accesses to SiteFinder content and to look for trends. So there are some difficulties. How exactly are we going to get the necessary data. I'll suggest one method that I've actually used so far.

There are some privacy concerns, to be sure, that can be addressed, or at least I think and hope they can be. Anything a user is doing that's a little bit out of the ordinary, be it using a non-default name server, not using the name server they received, for example, by DHCP but some other name server could confuse the analysis, users who specifically seek out SiteFinder content rather than by who receive it by making typos. Certainly I specifically requested SiteFinder a number of times in order to make screen shots, for example, or to probe it generally.

And also, these inferences will be strongest for large networks. The smaller your network, the less data will be available in any sample that I might be able to obtain and so the harder it is to measure exactly what is going on. Here is the citation to the article that I published just last week, actually, on this very subject. For those who haven't read it and are interested in analysis of who's viewing SiteFinder and who is not, this might be of interest. There's a company called Alexa, alexa.com, that has been kind enough to offer me data. Backing up to show how they got the data, this is a screen shot up above, it collects data about what specific URLs users end up visiting. So if you go to msn.com and you have Alexa installed, it tells the servers that you visited msn.com. This is helpful for analyzing how many Internet users go to what web sites, tracking trends and popularity of all sorts. And it can be used to track SiteFinder. This isn't a completely perfect approach. There are some representativeness problems, whether Alexa Toolbar users are actually the same as ordinary Internet users. But nothing insurmountable, at least to the best of my knowledge after talking with Alexa staff and looking at the data itself. So they were kind enough to send me some files that looked generally like what's shown at left, data about the network block, that's a slash 24 address with the final low order byte removed, the date and time of accesses, and the specific URL accessed. They don't show me the URL parameters - anything after the question mark. They do show me the base URL. They sent me this information for all requests from verisign.com from the beginning of September through September 29, and that's the last batch of data that they were kind enough to send me so far. I believe they'd send it to others who have a bona fide research interest, and they didn't make me sign anything saying I wouldn't pass it on to anyone who wants it, so if you want it in order to perform your own analysis, just send me an e-mail. Just backing up for a second, we can look at the IP addresses here and begin to draw inferences. For example, we might figure out, say, from IP WHOIS or from routing data, where China is on the network, what kinds of IP addresses are China and are they looking at SiteFinder or are they not. The idea latent in the background is of course in China in some instant somebody mistypes a domain name. It goes without saying, with hundreds of millions of Internet users, people are making typos every minute. So if we see a drop-off even among the Alexa toolbar users as steep as here from the 23rd to the 24th, something has happened. What has happened? It must be that china, through their various backbone filtering systems, actually, has decided not to allow accesses to the SiteFinder content. And then by probing China's network in various ways I developed previously, we can attempt to determine just how are they doing it. Are they doing it on the basis of the domain name or packet sniffing or the IP address of the SiteFinder server? In any case, we can figure it out and determine to really a near certainty, yep, from these many days of zero requests to SiteFinder from users with IP addresses in China, China, taken as a whole, decided not to allow use of the SiteFinder network. There are others, too. Here is Adelphia, a ISP in the mid-Atlantic region. They disabled SiteFinder on September 19th, so they had only a half a day's accesses to SiteFinder on that day followed by four days of very low access and then it starts to come back up, ultimately goes back to a higher level consistent with SiteFinder being accessible from the beginning. So what do we see here? The only inference I can draw from this data is that someone at Adelphia decided to turn off SiteFinder, that is to disable it for their users, and then later someone else, maybe the same person, had a change of heart and turned it back on. And you can tell any of a number of stories. Maybe the technical department turned it off and the legal department turned it back on. I don't know.

I just made that up. But certainly that's the kind of thing that one immediately has in mind when looking at this plot. now, we're going to want to aggregate this data. We're going to want to stack these graphs on top of each other, essentially, in order to draw inferences as to the Internet as a whole. It's all well and good to talk about Adelphia or AT&T or China, but really we want to know about the Internet taken as a whole so we're going to want to stack those sorts of plots one on top of another, vertical summations. Now, just thinking about the theory here for a moment, what are we likely to see? If it turned out that no one had ever disabled SiteFinder, we'd expect to see a plot like this.

First, there are relatively few people going to Verisign.com on any given day, that is making web page requests for content on that domain.

I'm sure some hundreds of thousands, but in any case it's not in the tens or hundreds of millions. then one day, SiteFinder is introduced.

We see a large spike, and a plateau that remains until, I suppose, SiteFinder is disabled. So this left plot is what we would expect to see if essentially no networks were disabling SiteFinder. And then there would be a drop-off, unfortunately not pictured in my now out-of date idealization. If some sites decided to block SiteFinder, we'd see drops, drops here, another one, another one, each of these reflecting one ISP and some quantum of traffic no longer being received by the SiteFinder servers. If that's what we were looking at we'd expect to see drop after drop until, well, one day we might get the big drop where SiteFinder is suspended. Here's Alexa's ordinary public rank chart, the sort of thing they tabulate every midnight pacific time as to every top-level -- sorry, every second-level domain their users have accessed.

We can see there's an ordinary weekly cycle. Verisign gets more traffic on the weekdays than the weekends, much like the other web sites, and then a spike at the introduction of SiteFinder, then a plateau, and then a drop when SiteFinder is disabled. Is this really a plateau? No, it's not. There's some data at the beginning where it's growing slowly. I'm not sure what to make of it but if you thought SiteFinder might be under attack, like a denial of service attack like a name being removed from a zone file, you'd expect to see something like this where SiteFinder was operating at something less than 100% because it was under the load of the denial of service attack. Or if the web servers and their connecting links weren't capable of carrying all of the traffic and so there was this little bit of missing traffic just where my mouse pointer is, also you'd expect to see a further growth as Verisign staff managed to get that problem corrected. Then we definitely see a noticeable drop here. You can see that the blue line touches the black line at the top end of the graph, but then retreats down from it.

It's not a very large retreat. On the other hand, this is a graph with a logarithmic "y" axis, which is to say every time there's one of these big markings on the left that's ten times as much as the marking just below it so this reflects something like -- it's about a 10% drop-off in traffic, actually, when you do out the arithmetic of where these markers are spaced. That, as it turns out, is consistent with the kind of estimates that you'd get from thinking about how many Internet users are in China, how many are at Adelphia and elsewhere on the Internet where there is evidence that SiteFinder was disabled. So that's what I take away from the ordinary public Alexa rank chart. Here's my chart prepared based on the data they sent me. It's a little bit of a disaster for which I apologize, but we can understand at least some of it with a close look. The solid -- I'm sorry. The dashed line here in black reflects total traffic to SiteFinder. No one has yet been able to explain, myself included, what exactly is going on here on the 23rd with such a large drop in traffic. The best guess is that there's something wrong at Alexa where Alexa's data collection systems failed. A possibility also is that SiteFinder was somehow not working properly, although if that were the case I bet we would have heard about it some way other than by me making this chart. But putting that aside, we can see that traffic generally is relatively consistent, actually, at roughly this level between what is labeled at 70 versus 80 on the graph.

Although there must be something like the word thousands missing, thousands of hits or tens of thousands, hundreds of thousands, at least as to this black line. I think I divided everything by a thousand for the aggregate line, but we certainly should straighten that out before we try to draw too much of a conclusion from the dotted line. there's lots more that could be done, of course, if we had data from some better toolbar system, for example the Google toolbar which is installed by so many more users, even in this room, I'm sure, than the Alexa toolbar, it would be possible to draw much more rigorous conclusions about, for example, smaller networks. As it stands, many networks are too small to have one or many Alexa users, and so it's impossible to say with any real certainty whether they did or didn't disable SiteFinder. On the other hand, if we had more data from probes, more Google tool bars than Alexa tool bars, we would be able to draw inferences about other networks. I've talked to Google about it. They said basically no to a first approximation. They're super paranoid about privacy concerns as to their toolbar and didn't want to give me this data. It's not clear they even retain the data and certainly it would be possible to design a system that doesn't, so to be honest I'm not hopeful. On the other hand, Verisign could produce this data and run the analysis itself or delegate the analysis to some outside organization. Not necessarily me. In fact, perhaps ideally not me. I'm busy with other things and we might want to just hire a paid consultant here. But there's lots that we could figure out from looking at the log data, figuring out who used to be asking for SiteFinder and then stopped; figuring out who asks for lots of other content but never asks for SiteFinder; and inferences that could be drawn as to exactly which networks, then, are blocking SiteFinder. Before I close, and I know we are running late, I wanted to present one other aspect of this research, which is figuring out which pages on the SiteFinder site seem to be of greatest interest to users.

Because I've got data from Alexa as to which specific URLs, including the file names, were viewed, I can draw conclusions as to how many people clicked on, for example, a popular category, what proportion of users click on popular category links. I can't tell when a user clicks on a "did you mean" link because that is not observed by the Alexa Toolbar. I think there is a recording of that fact by the SiteFinder tracking systems. I don't want to swear to it but we could certainly find out quickly enough. In any case, it doesn't come out in Alexa data that was sent to me. They only sent me data about verisign.com, and I think that goes through a third-party domain name. So here's what I can tell you. At least as among the Alexa Toolbar users, the right column gives the proportion of accesses of page views that were each of the URLs designated in the left column. So we can see that a little bit more than 80% of page views on the SiteFinder site are for the /lpc page, the default SiteFinder page which is what we've just been looking at here. some of these other pages are actually of great interest. What proportion of users read the terms of use, which at least if you take them at face value would purport to bind everyone who visits SiteFinder. We can see that .15% of users see the Terms of Use, 81% see the main content. So that's, what? About one out of every 500 users reads the Terms of Use. I don't know if that's a big number or small number. We could ask amazon what proportion of its users reads its terms of use. I'm sure it would also be a number on the order of one out of 500. In any case, there are some lawyers I know already thinking about the question of whether the Terms of Use are enforceable, and so here's some data that might be of interest to them. As I said, I don't have any data about clicks on the "did you mean" links, but I do have some additional information to offer as to the categories. When a user starts looking at the related categories or the suggested categories, the user is likely to make many page views, such that this statistic here, 16.19%, actually somewhat overstates the true proportion of user sessions that include a page view of the categories. After all, if a user looks at any categories, they might look at two or three or ten, which would tend to pull the average up misleadingly. When you account for that, as best I can with the data that I've got, it looks more like 12% of user sessions include at least one popular category view. Even that is probably an overstatement since I have to count each /24 network as its own user when in fact, you might well have multiple users within a /24. Well, that's what I've been able to do with the data so far. I'm sure there's more that can be done with a bit of creativity, and I welcome suggestions and, of course, questions.

Steve Crocker: thank you. I'm going to jump in myself because I'm keenly interested here and ask a question, and then back off and let others.

Could you back up to the slide that shows the sharp rise and then bouncing along the top before that? That one. Two questions, one large, one small. The small one is the "y" axis looks like the numbers are going smaller as you go up. I'm not sure what that means. But to the main point, you've got a substantial variation on a weekly basis well before the direction, and then you suggested that at the top, even though it looks kind of like a square wave, there's some potentially interesting secondary effects up there. And it isn't evident to me that those effects are actually as large as you are suggesting when you compare back to the previous weekly variation; that is, on the sort of last part of the rise, is that -- I can't quite tell which day of the week we're talking about, but that could be related to the same kind of variation that takes place on a weekend, and then that variation along the top is -- even if it's logarithmic and 10%, it's still modest in its variation compared to the variation that's taking place anyway prior to the introduction of SiteFinder. so I'm --

Benjamin Edelman: got it. So first, as to why the "y" axis is getting smaller the higher you go, the important word here is "rank." This is the way Alexa prefers to show its data. They have the number one site on the Internet as measured by their ranking methods, number three site, number three site, so a smaller number indicates more traffic.

That's the way we did it. We're using their data.

Steve Crocker: those aren't traffic numbers. Those have to fit into some larger pattern to guess what the numbers are and one might imagine a hyperbolic function that --

Benjamin Edelman: it's a pro rata distribution. I've been working on algebra to figure out what it all means --

Steve Crocker: if you can give us what those numbers are as opposed to giving us the rank.

Benjamin Edelman: they can.

They're busy folks and we don't like to bother them too much. They're working on it and don't like to be bothered too much. Happy to send me data, don't like to answer questions.

Steve Crocker: do with it as you will.

Benjamin Edelman: As to the second question as to weekly effects, as it turns out the fluctuation here is it would be much smaller if you stacked it on top of more data and then you watched it fluctuating on a site with very high ranks. What's a good analogy here? If I were to put five grains of sand on the table in front of one of our computers, we'd notice. There's no sand on the table now so five grains versus ten grains would be a big deal. On the other hand if you go to the beach and put five or ten grains of sand on the beach obviously it's no big deal because there are millions or billions already.

Steve Crocker: let me see if I can translate back into the terms you put it here. A variation of the size that showed up in the weekly thing wouldn't be enough to knock you from number one to number two if that's all that was going on up at that range.

Benjamin Edelman: precisely. It just wouldn't work that way. But you're right to be mindful of it.

Steve Crocker: just practicing taking the quantitative reasoning exam. That's all.

All right. others, on the phone or here.

Mark.

Mark Kosters: actually, this is more of a statement than a question, but I just -- I like to commend you for actually doing this sort of research and I hope that many more in the community do these things, too, to actually assert -- give some real credibility behind some of the assertions that have been made.

Benjamin Edelman: thanks to the Berkman Center that funds me, to the extent they do, of course, in doing this work.

Steve Crocker: that's a pop-up ad, I think.

(laughter).

Steve Crocker: anybody on the phone?

K.C. Claffy: this is KC from Caida. I'm just confused on slide 16.

You've got these rows and columns, and it says 16% of people -- 16% of accesses in the second row. Can you clarify, am I misreading it to mean that 81% of the people who land the page, at the main page, don't actually click on another link, don't actually use it and only 16% use the second link or whatever, and then on down? How am I supposed to interpret that table?

Benjamin Edelman: I wish you could draw those sorts of conclusions. That's not quite what the data says, unfortunately.

The data is related to the sorts of questions that you ask and we can use this data to at least put bounds on the number of people who do or don't look at other pages on the SiteFinder site, but we can't crisply say what --

K.C. Claffy: can you give me those bounds?

Benjamin Edelman: sure. Here's one that we can tell you.

At least one out of 500 that use in the /lpc referral to SiteFinder actually results in the user subsequently viewing the Terms of Use. And we can say that because there are only a certain number of views of the Terms of Use to go around. If a million people looked at the /lpc main results and 10 people looked at the Terms of Use we know that at most one out of 100,000 looked at the terms of use. Because we have no information on the clicks on the "did you mean" though, because that's what we imagine most users are likely to click on, it is difficult to say what proportion of users exited from SiteFinder without clicking on anything at all. There, we really should get this data from Verisign. I mean, they've got to have it thanks to the web bug and the pages and the various tracking methods that they are using, they say they're using that we can see they're using. This is a question that Verisign ought properly answer and ought to be in a position to answer.

K.C. Claffy: Did you ask them -- have you asked Verisign for that data?

Benjamin Edelman: I have not asked them for it. that's a great question, and since I asked them for something else, I certainly had the opportunity to ask them for this, too but I didn't think of it at the time.

Steve Crocker: just a second.

The second line there, could you just remind us what that, the /spc, what that refers to?

Benjamin Edelman: it refers to two things, actually. One, if you click on one of these popular categories, a page view of /spc will result. Two, if you type something in to search the web and press the "search" button, a page view of /spc will also result.

So this says 16.19% of Alexa's traffic was one of those two features of SiteFinder, the searching plus the popular categories. When you back out the number of people who make multiple searches it would seem to be on the order of 12% of user sessions that include one of those two features.

Steve Crocker: and so what can you say about the relationship of those first two lines to each other which --

Benjamin Edelman: right. Well, we certainly know that not every person who looks at a main SiteFinder results page clicks on something that would trigger a /spc because then these two would have to be equal. Instead, it has to be -- suppose we adopted the assumption that a user views at most one /spc result. Either you look at one search result or zero. Either you click on one popular category or zero. Then we would know that 20% of users use those two features taken as a whole.

Steve Crocker: I think that's what I was tending towards.

So -- right. so it's the ratio of those things. and with the caveat that that's if there was only one lookup, subsequent lookup, after you got to the main page.

Benjamin Edelman: right. Fortunately, these are I guess the same kinds of problems that ordinary webmasters have to deal with when looking at their web analysis, if they don't have more sophisticated tracking methods. So if you have a webmaster that has to deal with traffic to your site, they can probably deal with this quickly.

Steve Crocker: so I've got a mental model of a state diagram where you have the first second and do or don't go on to the second state and the probability of going onward, at least with the assumption you have talked about, is 20%. And then this 81% is a derived number when you look at the whole thing and the sort of underlying number, that 20% transfer, if you will.

Benjamin Edelman: yes. That's what I was attempting to get at here after correcting for the fact that some users look at multiple pages in the second state.

Steve Crocker: all right. thank you.

Sorry. K.C., did you have more? I'm sorry to interrupt you.

K.C. Claffy: I'm just trying to get some consistency between this data and the survey data that Verisign has articulated, but I'm unable to do so I'll just wait.

I think getting the log data out of Verisign would be good if Ben is willing to take on the work. Apparently they'll sell it to you, Ben.

Benjamin Edelman: I hadn't heard that.

Certainly there's no particular reason to think that the survey data would be specially correlated with the actual log analysis. After all, the survey data asks people in the abstract about what they like and what they don't like, and this measures what they actually do. There's a large body of literature saying that the two things don't have to be perfectly correlated. Of course, dentist, I always floss my teeth before I go to sleep. And then we know that people don't do that. But putting aside that kind of a problem, you would expect a general correlation.

And, you know, maybe we're seeing a general correlation here. A 20% click-through rate or a 12% click-through rate, those seem like big numbers to me, actually. And so that might be taken to reflect a very positive response to SiteFinder. I don't mean to say that 12% isn't a big number. In some circles, certainly it's a very big number.

K.C. Claffy: well put. Thank you very much.

Steve Crocker: any other questions?

Rick Wesson: a question. I'd just like to ask Ben how long he worked on this and what kind of time it took. I don't think there's many people that can go out and do this kind of analysis, and I wondered if he could comment on it.

Steve Crocker: thank you. and is that Rick?

Rick Wesson: yes, sir, it is.

Benjamin Edelman: I think the total time in days spent working on this was on the order of seven to ten days. Some of the days I got to take as vacation days; sometimes I get to leave my apartment and go for a run or even go to an amusement park and go on some roller coasters. I did both of those things during the course of preparing this study.

All told, I would think it's something like a hundred hours of work, 80 hours of work. Ultimately, the data speaks for itself, and it only takes a certain amount of massaging to make some charts out of it. Even then, more time was spent trying to make sure the axes on the charts were properly labeled than instructing excel to make the chart and export it to a GIF. What's necessary here, I think, is a willing provider of the data, someone with basic data analysis skills who knows enough about the facts on the ground to know what questions would be helpful to try to answer, and then a place to publish it, a web server somewhere. I'm fortunate in that those things came together for me but I would really love to see some other people doing this also.

Steve Crocker: I haven't left my office at all and I feel like I've been on a roller coaster.

Anybody else?

All right. well, Ben, thank you very much. it's been enormously helpful. I appreciate, again, the effort to come down and present this. It's certainly been consciousness raising for me and I'm sure for others as well.

We move to the last formally scheduled item on the agenda, a statement from Global Name Registry, and it's my understanding that they're going to read that over the phone, is the present situation. Right?

>>>: yes, that's correct.

Steve Crocker: who do we have on the phone?

Hakon Haugnes: my name is Hakon Haugnes, and I am the president of the Global Net Registry. and I am lucky enough to be able to present our statement by phone. We had prepared a video statement for the benefit of the committee but unfortunately, that couldn't make it. So I would thank you, for the ability to present this here.

Steve Crocker: let me thank you very much for joining our meeting, as you are. And we can hear you quite clearly, which is good. Let me ask you to read slowly so we are able to capture what you're reading very clearly.

Hakon Haugnes: no problem. Feel free to stop me any time.

The reason we wanted to present this today is that we have had growing concerns following the community reaction to the change in the DNS operations for dot com and dot net which related to certain workarounds for package. The SECSAC has already pointed this out to the board on September 22nd. As a result of these workarounds, there is a real threat for the users of the top-level domain space, and they may already be experiencing difficulties and outages. I didn't have the chance to do so earlier but I would actually commend Ben for his research, and he has, in fact, pointed out ISPs that we didn't know about that has implemented this patch. That has resulted from what we think was a hastily introduced patch to the BIND software including the so-called root delegation feature. A number of operators may have already implemented this patch and listed some of them, and the incorrect use of the software will impact dot name second level users. A recent misconception about the cause of this issue is that the dot name TLD returns a wildcard or star record response. That is not the case. The dot name gTLD works only by returning nondelegation resource records for the second level. For example, MX records for all shared second-level names.

This was contemplated in our ICANN contract. And these valid and vital resource records could now be replaced with an MX response. In other words, anyone who configures the dot name zone as delegation only or fails to exclude dot name from their root delegation only configuration is currently blocking all e-mails to any address of the type first name and last name dot name. To be clear, the only users that are affected by this are users of the dot name second-level e-mail forwarding service.

The third-level domain names are not affected, but even if this affects only a relatively small number of Internet users, it is a large portion of our users that could be potentially affected, and we do take this loss of service for our users very seriously. So accordingly, we wanted to present today to respectfully ask the relevant parties for the following three actions to be taken in this matter. First, we would like to ask the Security and Stability Advisory Committee, as a matter of urgency, to issue an advisory warning about the destabilizing effect the current BIND patch has on the Internet. ISPs and other DNS server operators should be warned to remove all use of the root delegation only functionality of the current BIND patch and use the delegation only functionality on a per TLD basis only as strictly needed. Second we naturally appreciate the value of the ISC in bringing to the Internet community its BIND DNS server software. In fact, we would like to thank the ISC for already updating its web site with a notice to exclude dot name from the root delegation only functionality. However, we would respectfully ask the ISC to withdraw the root name only functionality in its current patch as this affects the stability of a number of TLDs, including dot name. If the root delegation only functionality is not removed from the BIND software, DNS operators will have to exclude from the configuration an unknown number of current and future TLDs that may return nondelegation resource records from their authority zones. In other words, it becomes far too easy for DNS operators to block far too many TLDs inadvertently. Asking DNS operators to actively exclude TLDs (scribe note: dropped text) Third, we would like to ask all DNS server operators to discontinue the use of root delegation. (scribe note: dropped text) the delegation only functionality should be used.

We have established a separate e-mail address for ISPs and other DNS operators who would like to get more information on how this relates specifically to the dot name, and we would like to ask such operators to contact us on delegation-only@gnr.com. Finally, we would like to thank all the relevant parties in advance for an immediate response to this threat to both dot name and other TLDs, and we would urge the Internet community to take this experience into account if implementing similar changes to the Internet in the future. I would like to thank you again for the opportunity to present to this committee, and open up for any questions there may be.

Steve Crocker: thank you very much.

I think that's informative on a couple of levels.

I think there's a couple of questions and maybe some responses but I would like to exercise my prerogative to ask one or two questions. Have you actually observed problems and do you have any measurements on that and that's one question. And a second question is has it been -- how hard has it been for the users involved or for whoever is serving those users to figure out what the issue is? That is, has the effect that you're describing been clear or has it been hidden.

Hakon Haugnes: on your first question, we have been alerted of this issue by a couple of means. The first one was a mail to the NANOG mailing list by a company called Outplace, and a company which is operating a very large number of e-mail addresses on the Internet today.

We have subsequently been trying to gather our own data to try to find out what the extent of the problem is, but we haven't so far been able to draw any conclusions except that there seems to be some effect, but some of our data is almost like Benjamin Edelman's. We can't really explain it because we haven't had enough time to look into it.

Steve Crocker: if you could supply the real data, it would be helpful.

Hakon Haugnes: we can absolutely make data available to the community. So we may get help in drawing conclusions.

Steve Crocker: okay.

Hakon Haugnes: and we have also been testing the BIND patch on the ISC's own servers and we have concluded it would lead to a response that would lead no non-functionality of the second-level dot name addresses but it now has been updated and it now works.

Steve Crocker: good.

Hakon Haugnes: did that answer your first question?

Steve Crocker: well, I was concerned about whether -- when people have a problem, do they know where that problem is coming from, or is it just things don't work and nobody knows why, sort of a confusion factor.

Hakon Haugnes: we don't know the exact answer to that question, but it seems plausible that for end users seeing their e-mails bounce to otherwise valid dot name e-mail addresses would not have any idea why this is happening and would likely complain to their ISP or registrar, who may not even have the full knowledge about the issue.

So I think it's fair to say that we don't know the scope of, or the extent to which end users are able to distinguish the reason for the fault, but hopefully we'll be able to gather more information on that in the very near future.

Steve Crocker: good. We're fortunate enough to have Paul Vixie right here, so when matters of urgency relating to BIND come up, my thoughts first turn to him.

Would you like to say something, Paul?

Paul Vixie: I can reiterate what I said at the meeting last week, which is that the configuration that we publish on the web page and in the examples is the configuration that we run on our own servers. We make no recommendation that others do likewise.

The presence of these two features, one of which is called delegation only and one of which is a longer name, root delegation only, is due to pressure from our user community. And in fact, the existence of these features does not exactly represent ISC's policy. We are using them now that they exist, but they would not be there except for user demand, for both of them separately, and we would not be removing either one of them.

Steve Crocker: all right. Thank you.

Mark Kosters: actually, I have a question for Paul. That is I'm wondering how you assess the sort of user request and if that information is publicly available in some form.

Steve Crocker: I like that question, Mark.

(laughter).

Paul Vixie: which form of user request are you referring to?

Mark Kosters: well, the basis for both delegation only and root delegation, whatever the second request is, was based on user feedback; right?

Paul Vixie: yeah.

Mark Kosters: and I'm just wondering how you assess that and if any of this information was available for others to look at.

Paul Vixie: any of the comments that are made in public forums, like the BIND users mailing list or the equivalent usenet news groups or any of the operational forums, like NANOG or protocol forums, like IETF, are, of course, publicly available.

The stuff that comes to my personal in box is not publicly available, and I'm not interested in republishing what was originally private e-mail without permission.

Steve Crocker: any other comments?

All right, Hakon thank you very much for the presentation. I think I'm going to move on to the final part of our meeting today. I appreciate your time.

Hakon Haugnes: thank you very much.

Steve Crocker: good.

We're at that point where we used -- whose phone is that?

I hope it's not mine.

Sorry about that.

Go away.

I know.

On the record.

I took the battery out.

In principle, this should be a period where we raise any other issues that we have that need to be taken up beyond the ones that have been discussed. We're a little short on time. I think I'll allow about 17 seconds for anybody who raise something quickly if there is urgency and necessity to do that. Counting one... two... three...

All right. Time's up. Good.

Everybody I think is keenly interested in what happens now. I'm certainly interested in what happens now. We're -- we, the Security and Stability Advisory Committee is going to attempt to write down what we have seen, marshal it into something readable and reach some degree of consensus amongst ourselves. We're also going to share trial, ideas and our preliminary thoughts with some selected set of reviewers, specifically we will include Verisign, of course, but others throughout the community. We will obviously face a convergence challenge as to whether or not we can reach a sufficient level of agreement and consensus on what we're going to say. And that I think that burden will fall heavily on me to make that process, converge one way or another.

When we're done, we will issue a document that has what we've seen is explains things as clearly as we can. I do not want to prejudge what we're going to say or commit to whether we're going to make specific recommendations. My preliminary thinking is that it will be useful to divide the discussion -- to divide the discussion along the lines of a process issues about the introduction of such services, versus the fundamental role of such changes in the larger architectural space. But even that I'm not sure whether or not all that is going to remain when we go through this process. Our target audience is ICANN itself, the management and the other constituencies on the board, and it is the technical community and there are obviously technical questions that have been raised here that require consideration beyond the limited resources that we have within this committee. And then last but not least by any means, our audience is also the general community that is looking at these issues from every perspective from, a process point of view from, a technical point of view from, a policy point of view. And we'll look to try to learn at least as best we can help them what is at issue here and what is not. The further challenge in this process is that our charter is security and stability and there is not a crisp definition of what is or is not within that charter. Clearly if a machine stops working one can say, that was unstable. But there are many other much broader issues that are arguably within stability and security issues, but that others would argue are not, that they're outside, that they're some other thing. As I've said before, rather than try to sort that out in a definitive way or make a decision that errs too that are on one side or the other, my intention, I'll use person singular here because I don't know how well this will hold up. But the formula I think will be useful is that we'll try to write coherently as if those broader issues fit within our charter, but then very carefully label that that's what we're doing and leave it to the various audiences that we're addressing to decide what weight to give to that and to -- how to fold that in to advice and information that they get from other sources. And the fact that we are an advisory committee as opposed to a decision-making body, I think I have grown increasingly happier and more comfortable knowing that we don't have that final responsibility and weight on us and we will cheerfully and willingly pass the ball to others. Are there any final questions?

Any comments that people want to add before we bring this meeting to a close? All right. With that, I want to thank everybody for coming, quite a lot of work has gone into this meeting as it was into the last meeting. Marilyn Cade and Carla Lafever put a lot of work into the logistics, the ICANN staff and fabulous transcription team over here, the speakers some of whom traveled some distance. And the committee for putting in many hours of time. And of course the audience for coming and helping out and I particularly want to thank Verisign for putting together a substantial package of information and for bringing the full team here to interact with us today. Thank you very much. This concludes our meeting today.

Comments concerning the layout, construction and functionality of this site
should be sent to webmaster@icann.org.

Page Updated 17-Oct-2003
© 2003 The Internet Corporation for Assigned Names and Numbers. All rights reserved.