SECSAC Meeting
Real-Time Captioning
7 October 2003

Note: The following is the output of the real-time captioning taken during the Security and Stability Advisory Committee meeting 7 October 2003 in Washington, DC. Although the captioning output is largely accurate, in some cases it is incomplete or inaccurate due to inaudible passages or transcription errors. It is posted as an aid to understanding the proceedings at the session, but should not be treated as an authoritative record.

Steve Crocker: good morning.

Sit down. Oh, I get assistance here. Excellent. Oh, my words are being recorded. Thank you all.

I have a number of objectives for today, but first and foremost is to get through the day and to do it reasonably closely to the schedule that we've set. So I'm not under any illusion there will be people straggling in over the next, I'm sure, minutes or longer. But I think it's appropriate that we get started and move right along.

My name is Steve Crocker. I chair ICANN's Security and Stability Advisory Committee. It's taken a year and a half of practice to be able to say that mouthful in one breath. I will introduce ourselves a bit more after -- in a little bit. We have members of the committee sitting at the panel. We have other members joining us by telephone. But I would like to start off by turning the floor over to Arnaud De Borchgrave, who is a senior director here and a director of the Transnational Threats Initiatives and has an incredible resume that would make it clear that as important as the issues are we're dealing with, they fit into a much bigger world that he has written, breathed, and lived his entire life.

Arnaud De Borchgrave: on behalf of John Henry, a warm welcome to this bilingual city, where truth is a second language.

(laughter.)

Arnaud De Borchgrave: seldom spoken, however. As you heard, I'm director of the Transnational Threats Initiatives but I've been a journalist now for 67 years, and you know what journalism is all about. We explain to others what we ourselves do not understand. But, seriously, since I lived for three-quarters of the last century, just think of the time when the Germans were dropping bombs by hand over allied trenches in World War I and then move forward to when we dropped the big one on Hiroshima. That was only 20 years. That's how fast things went in the previous century. If you look at the first manned flight by the Wright brothers and the landing on the moon, that was only 66 years.

Imagine how much faster things are going to be moving now in the 21st century.
In less than ten years, everything has changed with the Internet, which you are, of course, intimately connected with. It's changed the way we think, the way we deal with our friends, businesses, the way we conduct our lives. But very few political leaders, unfortunately, seem to grasp that we are living through a period of social change that is more awe-inspiring, I think, than anything we've seen in 5,000 years of recorded history.

So I just want to say that on behalf of CSIS and the program that I direct, I think we should also bear in mind that it is now glaringly obvious that there is a growing abyss between the political, economic, technological, scientific, geostrategic, geoeconomic, and psychological knowledge of the masses and the people who represent them. That's on the one hand. And on the other hand, the knowledge that is indispensable to arrive at rational, logical, and moral conclusions.

So unless you've made other plans, I hope you have a nice day. Thank you.

(laughter.)

(applause.)

Steve Crocker: Arnaud, thank you very much.

I look to him on advice in all matters.

And as we walked in, he pointed out one of the important things to keep in mind, which is there are very many places to eat and sandwiches available across the street.

For all matters great and small, thank you, Arnaud.

Now, a piece of stagecraft here. How do I hook up my own machine?

John Crain: you're going to have to go to the podium.

Steve Crocker: I go over there.

All right. Good.

So, once again, welcome to ICANN's Security and Stability Advisory Committee meeting. This is an unusual kind of meeting, not one that we've ever had before. I appreciate everybody's participation, the people who have helped put this meeting together, and all of you who have come. The -- and I'm sure I will forget any number of people in the acknowledgments, but I will give it a shot.

Before we get into the agenda in particular, let me say just a word about what we are about. We are an advisory committee within the overall ICANN structure. We're composed of experts from different aspects of the Internet community who are focused on the domain name system, the address registries, and the research and security community. That worked nice. Nicely.

There are -- let's see, I'm a little out of order here. So let me just do that.

Here's the people who are currently on the committee. These names are all posted on the ICANN web site.

And as I mentioned, we're drawn from different aspects of the community. We are a -- fundamentally, a technical and -- committee, and the people who are on the committee have regular day jobs, not just to participate on the committee. We don't have people who are in the policy business or who are political members. No attorneys or bureaucrats or lobbyists, although half of us in the room probably are, including my good friend Marilyn, who is one of the best I've ever seen.

So I -- it's not that I don't enjoy those kinds of interactions. But that's not primarily what our committee is about. Within -- we fit within ICANN as an overall structure.

ICANN's charter covers the domain name system and addresses. Its fundamental purpose is to achieve coherence and stability of the operation. And it's an evolving process. The Internet has grown at an explosive rate. It has moved from a research idea to one that was broadly used in the academic and government community to one that became a commercial success, and one that is now practically embedded in our daily lives in every aspect and around the world.

That kind of growth brings any number of stresses, including unsolved or un-addressed problems before of governance and of the mechanisms in place to make things work smoothly. And so this is a learn-while-running kind of operation. And as I said, our job is to focus on the security and stability aspects within that overall larger ICANN process.

If one looks at ICANN from the point of view of where -- what parts of the overall organization are concerned with security and stability, there are perhaps four different participatory organizations. These are organizations within ICANN that are composed of representatives across the community, the Generic Name Supporting Organization, the Country Codes Names Supporting Organization, the Government Advisory Council and the Root Server Advisory Committee.

And then what I would call two specialist groups, the IANA, which administers the root database and address allocation, among other things, that is staffed inside of ICANN; and then our committee, which is staffed with unpaid experts from the outside.

What brings us here today -- let's see.

Well, before I do that, one other thing.

If one looks at who the other bodies are around the world that are concerned with aspects of the network and its evolution and its operation, here's one way to look at this. And this is purely illustrative. If I tried to fill up the slide with every organization, it would be completely dense and wouldn't convey anything except a lot of noise. So the most important word on that slide is the one in the top left that says "illustrative." But I have taken the opportunity to break this both by geography across the top there and by kind of layers as we see them from bottom to top, architecture and protocols and implementation at the bottom, up to the less-technical aspects of what happens when there is an incident, the response mechanisms, law enforcement, and policy and laws.

Excuse me.

Within this environment ICANN sits in a fairly broad way across all the geographies and across many of the levels. But, of course, in exchange for that breadth, it has extraordinarily little depth. It doesn't have very much operational control; it doesn't have very much legal control. It has a little, tiny bit. And it tries to knit things together across these -- the spectrum. But, primarily, in a cooperative and consultative way as much as possible.

Okay. What brings us together -- I'll get to the agenda in just a second here.

What brings us together is a -- an issue which is both specific and general. The specific, of course, is that roughly three weeks ago, Verisign introduced a significant change in its operation for the com and net domains returning an address of a host that was one of its own instead of an error code in the event that there was a query for a previously uninstantiated name.

Aha. So I forgot to make the announcement about cell phones. So please turn them off or silence them, if you would.

There follows from that a considerable amount of attention from a large number of organizations and individuals and companies about the impact that this was having, about the consequences. And there was considerable push-back from the community. Our committee, along with other organizations, took a look at the situation, and we asked first that this be suspended and rolled back to its previous state so that there be time to study the situation more clearly. And at first that was not what Verisign's response was.

ICANN management, not our committee, looked more deeply, issued a demand last Friday. And as I think everyone's aware, Verisign rolled back the service and suspended it on Saturday. In the meantime, we had issued a preliminary advisory which has a number of opinions in it. And it had three particular recommendations.

One of the recommendations was the very straightforward one consistent with what, essentially, everybody else was recommending, that the service be rolled back. But equally important were two other recommendations. One directed at the technical community, that there were obviously some issues in this region that had not been clarified and ironed out in detail; and that it was worth having the technical community look more closely at that. And then the other was a recommendation to ICANN itself to clarify the processes and try to work out the procedures by which these things could be handled in the future.

We also recognize that there was considerably more to this picture than we could document and perhaps even recognize on the fly, although there was plenty of information that made the recommendations that we made fairly straightforward. So we set about gathering more information, and we scheduled the meeting that we're here at today to bring that information out not in a final format, but as a sort of force-feeding the process, getting it marshaled, and providing a fast way of presenting it and acquiring more feedback and discussion about it.

In the past couple of days, with the pace of events, there has been a decision to extend this process a little bit more, to schedule a follow-up meeting to today's meeting, probably next week, digging in in great depth and providing a quite broad forum for Verisign to explain, present, and say a great deal more, and, equally, giving us an opportunity to probe more deeply.

At the end of all of this, we will report on what we have found and turn the process over to other parts of ICANN and to others. We will also make our report public to the community at large, which has been our style.

Let's see. What else do I need to say in general?

I think one of the things that I want to say about our committee is that it comes with a particular kind of burden in that by being drawn from throughout the community, it's not at all uncommon that we sometimes take up an issue for which one or more of our members has some involvement on one side or another. And today, of course, we have two members of our committee who work for Verisign; one who works for Yahoo!; another one who works for Melbourne IT.

Let's see.

And I've forgotten -- and there's at least one or two more that I have documented here and which we will document very clearly.

The larger point is that rather than -- since we're not a decision-making process and we're not vested with particular authority and instead come from primarily a technical background that operates by consensus, we've taken the very strong position that we leave the process open; we expect people to be forthright about their involvements and entanglements, and things proceed from there. And I must say that I think that process has worked quite well. If anything, individuals have been careful, more careful than perhaps they need to be in some cases, because they have their own reputations and the interactions with peers both within the community and elsewhere to worry about. So this has been, I'd say, a very positive experience in this respect, although not, of course, without some stress on each of the individuals involved.

All right. So, for today, we have an agenda that will start off momentarily with Scott Hollenbeck from Verisign's com and net registry, Director of Technology. Be followed by David Schairer, VP of software at XO Communications. And -- describing what was affected, what things have been observed. And then followed by Paul Vixie of ISC on community technical responses, including patches and reroutings. And if we are still on schedule, which I am determined that we should be, Richard Smith will talk on information flow. Richard Smith is a consultant from Boston and involved in privacy aspects.

We are providing coffee and juice for you on a continuous basis in the room here. And that's the end of the good news. You're on your own for lunch. And we've set an hour and a half break. There is any number of opportunities around the immediate environment. After lunch, Steven Bellovin from AT&T research will talk on protocol problems, and architectural issues, and John Klensin on Internet protocols and innovation. The time after that is for whatever other issues we haven't dealt with, for any issues that any of you want to bring up. And, if necessary, we can use some of the time in the wrap-up period that I have reserved at the end.

I intend for us to be out of here at 4:00.

The format for each of the talks is as follows: the speaker will have time to present his material, preferably without interruption. And then there will be time for questions afterwards. I will poll the people on our committee who are attending remotely, and then I will poll the committee members who are here physically. I will go back to the phone one more time. And then I'll open it up to the floor. And then we'll take it as it comes from there. We'll keep it as loose as we can and at the same time we'll keep it as -- moving right along.

The other boundary and ground rule that we'll observe is that this is focused on security and stability and it's focused on substance. There are many other issues of great interest and importance here. This is not the right forum for hassling about contractual issues or governance issues about ICANN or about any number of the relevant issues of the day. But there are forums for that, and if you need help finding them, we'll be happy to help you. But I suspect that most of the people here know their way through this maze pretty well.

Let me close with acknowledgments for the committee members and the speakers, who put a lot of time in, for the ICANN staff.

We've had volunteer help from Marilyn Cade, from Elana Broitman, Carla Lafever at MCI.
I've had really quite a great deal of help from Ari Elias-Bachrach and Paul Ott, graduates at George Washington University, helping to cull through the comments at the secsac@icann.org mail list and sort out the issues from emotional content, which there's been quite a lot, I can tell you.

We have invited Suzanne Woolfe and Michael St. Johns to help us in various respects. And both have been very helpful. Suzanne is on the phone, and Michael is sitting up here.

Let me introduce briefly the panel members who are here.

Michael St. John on the end, raise your hand.

Russ Mundy, Paul Vixie, Ray Plzak, Ram Mohan, Mark Kosters from Verisign, and Ken Silva. I'm just pulling Mark's chain a little bit.

And with that, I think that it's time to turn the podium over to Scott Hollenbeck from Verisign, unless, I suppose, in principle, there could be questions, but I hope not, of anything I said.

Scott Hollenbeck: thank you, Steve.

As Steve said, my name is Scott Hollenbeck, Director of Technology for the Verisign com/net registry.

Topic is Verisign SiteFinder.

As a quick overview, what you see on the slide in front of you is what we're going to be talking about.

Quickly describing what the service is, how it was implemented, I will briefly address some of the technical questions that we've been aware of over the last few weeks, I'll talk a bit about the need for guidelines for deploying wildcards in top-level domains, and finally opening the floor to questions.

I do have one comment or one request, please, that if you have any questions, please hold them to the end. We are on a tight schedule and I'm sure you will have some. So we'll deal with them all at the end.

Okay.

What is SiteFinder?

From the DNS perspective, it involves putting a wildcard a record in the com and net zones as described in RFC 1034. From there, the SiteFinder service provides web search assistance. On the service side, it attempts to match requested web sites with a known web site. The requested web sites in this case match the wildcard because they're nonexistent. The service also offers other search alternatives.

So if, for example, we don't seem to find a match, semantic or syntactic match for a nonexistent domain name, the web site would offer the user the ability to surf the web for something else or to search a list of fairly well-known categories.

For protocols other than HTTP, we provide a protocol-defined response.

In the case of SMTP, we provide an SMTP error message, in the case of others, there's a TCP or UDP error message returned.

I've listed a couple examples of the type of URLs that you could use to actually invoke the site-finder service. Unfortunately, none of which work right now., as a result of taking the service down on Saturday.

Okay, from a DNS perspective -- I apologize for the size of the font here, but you have to squeeze it in somehow. Before SiteFinder, if you were to issue a query to one of the com name servers for a nonexistent domain, you would get back an answer that returned a status of an -- our code of three, or DNS perspective is a name error. This means the name is not found.

After SiteFinder, looking at the bullet at the bottom, instead of an NXDOMAIN response, you received an no-error response and in the answer section, you actually received a referral containing an IP address to a Verisign server farm.

Okay.

The implementation of the SiteFinder service is based on a considered analysis of requests. And I touched on this briefly a moment ago. The service receives some information about a domain that's being looked for. And there's some semantic and syntactic searching that goes on to try to provide suggested domains and web sites that the user might have been trying to find.

For protocols other than HTTP, we do provide some kind of protocol-defined response so that the client application doesn't try to ping on the service endlessly. The details of the implementation are described in a public white paper that's available at the URL given on the screen.

Verisign conducted extensive testing prior to launch of the service for several months with a series of partners that were willing to try the service out, give us feedback for things that would work and things that didn't. Post-launch, we formed a technical review panel that consists of some of Verisign's engineering talent and some technical people from outside the company, with varying backgrounds, including name service issues, spam and e-mailing issues. And, finally, we have an ongoing monitoring program where we're, you know, soliciting feedback from the community, monitoring the service actively to try to stay on top of any issues as they arise.

One of the questions we've been asked fairly frequently is, well, just what kind of traffic are you seeing at SiteFinder? And if you look at the little table on the right, we have a chart showing 24 hours of analysis that identifies that 85% of all the connection attempts we've seen are for HTTP or SMTP traffic. Those are the top two protocols at the top of the chart.

For the other TCP protocols, instead of listening on the port and providing some kind of a server, we send a TCP reset error back to tell the client the connection can't succeed.

For UDP protocols, we return an ICMP port unreachable error. The client realizes they can't establish a connection and they stop trying.

Of that list of about 12 to 15 protocols, that only accounts for 97 and a half percent of all traffic, approximately. here are many different protocols, probably somewhere close to a thousand, that make up the remaining 2.51%, in varying percentages, all less than about one-tenth of a percentage point.

Now, what about the technical questions that have been raised?

I want to say emphatically that we are listening to the issues raised by the technical community, including the IAB commentary, the SECSAC message that was provided to ICANN. We've been listening in various technical discussion venues, such as Slashdot and the NANOG mailing list and have an active program on customer support lines, telephone support, et cetera. We are maintaining and updating a technical FAQ document available on the Verisign web site. In this we are collecting and publishing responses to some of these issues as they've been received and addressed by us. Earlier this morning, we released an extensive response to the issues raised by the IAB and SECSAC, found at the URL given on your screen. And I will speak to a couple of those issues in a moment. And I say a couple simply because we don't have enough time to address everything.

Beyond HTTP, the one protocol we have received a significant amount of feedback about is electronic mail and SiteFinder for electronic mail. A few days after launch, we provided an improved stub mail server to bounce messages received using a nonexistent domain. The server is there to -- where the ascending mail server would attempt multiple delivery attempts over time before it eventually gave up because it couldn't reach the nonexistent domain.

We're also considering deploying a wildcard MX record to provide a DNS name error response instead of the SiteFinder address. This has the benefit of returning some -- the name error to certain mail applications so they what I have in the way they did before SiteFinder was launched. f this works well, the SMTP response server can be eliminated completely.

Another issue that's been brought to our attention is the effect of SiteFinder on spam detection. One example of a situation that arose involved a dead relay black list service called dorkslayers.com. The impact was actually relatively minor and was resolved on 16 September. If anyone wants to go into the details of what that was or what was involved, I'll gladly answer that when we get to the question and answer section.

The point I want to make was the point was raised and it was actually addressed very, very quickly and very, very easily. Another situation that arose was where people said SiteFinder caused all kind of increased reasons for spam.

We did analysis -- spam message that is we've and collected over the course of doing spam research. what we found out is that many of the commercial spam services, even some of the free applications like SpamAssassin, and the spammers have pretty much given up on using false domain names and sending messages, simply because it's so easy to detect. Our empirical analysis shows the technique of using forward DNS lookups only catches about 3% of spam. And we're looking for more empirically based statistics.

Another issue that came to our attention was the effect of SiteFinder on misconfigured applications on the user, and on the administrative end of mail sending. There are situations where, for example, you have misconfigured MX records in your zone, or misconfigured approximaties in our user agent. These applications used to return an error code of 3, name error or NXDOMAIN in the buying world, which would provoke terminal failure in whatever program was issuing the query. Now, of course, this wouldn't happen if the misconfiguration used a wrong but existing domain, which would still cause a mail delivery problem or the nonexistent program was later registered.

And the issue that we have had with this particular issue is that it's been hard to size it definitively. What we have found, though, in doing an analysis have MX records in com and net subzones, of the approximately 20 million MX records for com and net, less than .1% of these records, 0.077% to be present, are misconfigured. So the problem is regularly pretty small in practice.

Privacy is another issue that we've raised. Verisign is not collecting or retaining any data for statements made in the privacy policy that was available in the SiteFinder website.

There's also an issue of the site being a single point of failure in an attack and being a dicey and enticing target to people who wish to capture information about individuals. Verisign has a proven track record for providing high-volume services, reliable. We have 100% uptime on the com and net servers. We invest more in that infrastructure than any other operator. We have a regular daily monitoring program to make sure that the facility was up and running flawlessly, not subject to attack. Providing reliable reasonable services. And, indeed, if SiteFinder were to go down, the service would produce time-outs or other error messages which while might be inconvenient, they certainly wouldn't be fatal or otherwise serious.

Okay, from an issues perspective, if there's anything else, as I said before, I'll be happy to take questions at the end. And we have an e-mail list set up where you can direct your questions via e-mail if you prefer to take them offline.

A little bit more.

All right, what about moving forward?

What about guidelines for top-level domains and wildcards?

Before launching SiteFinder we did extensive research to see what other people were doing with wildcards and top-level domain zones. The fact have the matter, they exist and they have existed long before Verisign did anything in com and net. As it says up there, we've found that wildcards have been deployed or tested prior to SiteFinder in several top-level domains, not just country codes. Dot museum, it's been tested in .biz and several.

Realizing that registries were doing it in inconsistent ways, myself and Matt Larson drafted a document describing guidelines for deploying wildcard in top-level domains. The document exists primarily to let registry operators know there are certain things they should consider doing to minimize some of the issues raised by the IAB, to make the service work better for end users.

As the bullet says, these guidelines describe strategies derived from extensive analysis. We worked on this for a good eight to nine months before launching SiteFinder. The guidelines describe ideas gleaned from comments received over the last year.

The last year going back to the IDN service that Verisign provided and the commentary provided by the IAB. We received commentary from the European top-level domain organization center through a couple of briefings we gave from them and public input received since the launch. We do anticipate working further on this document and we warmly welcome comments. Because we believe that consistent behavior among top-level domains would be a good thing for the whole community.

And this is actually my last slide. If you do have any questions, and if I can't answer them myself, I have a team of technology people sitting behind me and they'll be happy to answer any questions.

Steve?

Steve Crocker: thank you very much. Appreciate the briefing.

Let me start with committee members who are on the phone. Let me start by turning on the mike, and then let me start with committee members on the phone.

Thank you, Steve.

Anybody out there on the call who has a question for Scott Hollenbeck?

Let me insist that somebody out there, so we can test to see if the system is working.

Doug? Uh-oh. Actually, Doug dropped off the conference.

Suzanne? Jaap?

Jaap Akkerhuis: yes.

Steve Crocker: oh, good.

Jaap Akkerhuis: I have one question about SiteFinder. And I was wondering why it was not discussed before, in the working group.

Steve Crocker: if I understand the question, was why was there not advance notice and discussion in the IAMYAPF working group?

Jaap Akkerhuis: yes.

Scott Hollenbeck: I'll defer that question to Chuck Gomes.

Chuck Gomes: Thanks, Jaap, for that question. Very good question. One that we've heard a lot in the last few weeks.

I guess my first question, and I'm not asking you necessarily to respond to this right now. But certainly would appreciate some feedback, back to us in that regard. How an effective way, that could have happened.

I think that that's something that we're very concerned about. Going forward. And want to work with the community on. In terms of developing in best practices for introducing new services like this. We, as a company, absolutely do not want to cause harm. We want to be good citizens. I can tell you, we have had frustrating experiences with regard to introducing new services. And that is an issue that we want to work with the community in the future to improve.

But any suggestions as to how that might happen, I think there are a variety of issue that is come in. Certainly, it's very important for us to work together to minimize, and actually maybe a better way to say it is to maximize a service for everyone. Make it beneficial for everyone. We tried to do extensive, in fact we did extensive testing. We worked with a variety of company that is were willing to test with us, to try and identify problems. But we want to work with the community to develop best practices for this in the future.

And there are some challenging issues in that regard. But I think we can work together to make that happen.

Steve Crocker: I'm challenged here to try to stay inbounds. And yet feel compelled to point out that the ITF has been around for a long time, as has open a process as there ever has been. And I know for a fact that Verisign has no trouble finding its way to those forums. And is a welcome and active participant in many of the forums. I'm not sure I precisely understand the response that you are looking for guidance on how to do that.

Chuck Gomes: well, again, Steve, I don't want to go beyond the agenda today. Because I know this is on security and stability. But some of the issue that is come into play is the whole idea of proprietary information, competitive advantage, and new services. Bottom line, is that there are lots of people competing for things.

So I think one of the challenges that we have to deal with, isn't just a technical one. How do we work through the technical issues in forums like the IETF wihout getting so specific that we expose things to our competitors. Or another registry to their competitors or issues like that.

There are some things that go beyond the technical area, areas that I think we need to work through. I don't know that I have the answers for those. But I think we need to consider those as well.

Steve Crocker: well, all right, this does get to be interesting. But there is the quite obvious point there really are no competitors for a registry. A registry owns and completely controls the responses that come out of the domain.

>> well, I would say that we have over 200 competitors. That's just in the registry space. But also, as has been pointed out with regard to SiteFinder, there are lots of other businesses that are doing similar things. In fact, with SiteFinder, can continue to do those similar things.

That are also competing for the search business, for example.

Steve Crocker: uh-huh. Well, I think that's a helpful comment. I appreciate that.

Any other comments from the phone? Questions?

How about from people here on the panel. Mike?

Mike St.John: good. I have three question is want to address backwards from what I wrote down. Of the redirects to the SiteFinder pain, however-- what's the percentage of people clicking through on the referrals? On the referrals?

Scott Hollenbeck: we have those statistics. But someone on the floor has those accurately.

Mike St. John: can it be covered -- practice.

Scott Hollenbeck: just a point of clarification, with we going to focus on security and stability or usability?

Steve Crocker: that is an appropriate question. So it's tantalizing in a way to go down there. Is that related to a larger question on security and stability issues, Mike?

Mike St. John: put that way, I'm not sure. Let me think about it. I'll pass on the question.

Scott Hollenbeck: one of the things I offer, I'll glad to grab you through a break and walk you through the statistics.

Mike St. John: that's fine.

With respect to, you mentioned some issues have been reported, you mentioned maybe would be of them. Are there others?

Scott Hollenbeck: depends on what you mean by "others" relating to the ones I brought up and the IAB response?

Mike St. John: you said specifically that you had quote unquote been dealing with the issues as they occurred. You mentioned one or two. Were there others?

Scott Hollenbeck: yes. I think the IAB response spelled them out in fairly good detail.

Mike St. John: we did a good job of explaining what -- okay.

Scott Hollenbeck: yes.

Mike St. John: and the last one, actually, I think this is a security and stability one, on the matchal go rhythm -- well, I'm going to leave that one.

Steve Crocker: anybody is on the committee? Russ?

Russ Mundy: Scott, one of the things that has been a result of this, the use of SiteFinder in this manner, is it appears as though something that previously, or a number of things that were previously handled from a protocol response perspective closer to the user, are now further away.

These fall into some of the unexpected effects. One of the concerns that I have is how many things are there of that nature, or of the nature that gets close to the end user, that we haven't seen yet for changes in what is truly an infrastructure service, the DNS related pieces.

Have you folks looked at an analysis of what could be some of the cause of some of the other things that you've heard about, but are just sort of mysteriously weird responses to things that might or might not be related to SiteFinder? Are you looking at that?

Scott Hollenbeck: yes, indeed. that's been one of the agenda items for our technical review panel. We've been analyzing the issues described by the IAB and those that we become aware of in public forums, such as/dot or the NANOG mailing list. Some are anecdotal, some are real. We've been looking at those issues trying to categorize the likelihood of them occurring and if they've been documented to have occurred, what we feel the mitigation strategies are.

Russ Mundy: in terms of going forward from here, I would very much like to see some additional feedback from Verisign on things of that nature. And of the broader context of, perhaps, how one can make better assessments in the future of something, if x changes in the infrastructure, how broad, how do we determine the breath of the impact, especially closer to the user end. Use even though there's a lot of infrastructure pieces, there's ultimately some user machine somewhere that's reacting to these things.
Scott Hollenbeck: yes.

Russ Mundy: you do plan on another extension to changes.

Scott Hollenbeck: the output of the technical review panel will pretty much address exactly what you're describing.

Russ Mundy: okay.

Steve Crocker: any other questions? Suzanne Woolf on the phone, has sent in a message that she couldn't get in verbally.

And she writes, one concern raised among the engineering community is that any change with wide effects can be destabilizing, even if it eventually turns out to be benign. How would Verisign address that going forward?

Scott Hollenbeck: I'm not sure I understand the question. Are we talking about analyzing the impact of change?

Steve Crocker: well, that -- you're obviously in the position almost anything you do has a big impact because of the size. And so there are a lot of moving parts.

The question is how do you move forward and at the same time with abruptness, can have a destabilizing effect.

Scott Hollenbeck: I understand.

Steve Crocker: the question I think, if I have the question properly, how do you address that in going forward from here?

Scott Hollenbeck: in the specific instance of SiteFinder there are a couple of steps we've taken already. One involves the writing of the guidelines document and our continuing request for commentary and feedback. Secondly, we have established an external mailing list open to any member primarily of the technical community who is willing to join and help us discuss the issues in a rationale, calm manner. We are looking for commentary feedback while the site is down. Any factual based discussion provided by the community is going to be considered seriously.

Steve Crocker: good. If there are no other questions from the committee, I'd like to take a few minutes and open this up to the audience. Is there anybody in the audience who wants to frame a question for Mr. Hollenbeck?

Move to a microphone, identify yourself and provide the question.

Harold Feld, Harold Feld, Media Access Project. I Would like to ask how does Verisign define what constitutes a technical or security issue which falls within the purview of this committee and its own technical review board, versus a broader economic competition or policy question which would be subject to the ICANN process but would not fall within the context of its own technical review.

Scott Hollenbeck: I don't think it's Verisign's role to decide what comes within the purview of this committee, I'll leave that to Steve Crocker.

Harold Feld: but you have an internal process. How are you telling your internal review committee what to look at and not look at?

Scott Hollenbeck: we're looking at operational impact. Conformance with applicable standards. And that's pretty much it. We did months and months of technical research. In addition to market research.

And the technical research was really to look closely at the standards, first of all, to ensure that we were fully compliant with existing standards. That was a rigorous process. And I think going forward we will continue to listen to input from people in the community and the technical community in particular, with regard to the kinds of issues we're talking about today and deal with those issues.

If you have any specific technical issue that is you'd like to pursue, we will be glad to do that offline.

Richard Smith: Richard Smith. My question is about end user software. About SiteFinder was released how much testing of applications that are commonly used in the desktop did Verisign do? And I'm thinking of e-mail riders and other software other than web browsers.

In particular I'll give you one quick example, if someone misconfigures an e-mail server in Outlook and gives the wrong domain name, used to get a good error message. Now when you do it, you push the test message button to test the configuration, it says "bad e-mail address" as opposed to the SMTP server being misconfigured. I'm interested what Verisign that is done bee forehand to look at this broad application of application software.

Scott Hollenbeck: we have a large quality assurance department that analyze the product from a QA perspective prior to launch. The external testing that we did with a select group of partners was to assess the output on specific applications.

With respect to Outlook and the misconfiguration, that would be resolved by the deployment of a wildcard MX record. It would return it.

Robert Seastrom: hi. Rob Seastrom.

We have seen two standards for notice, one being the ex post facto of the actual implementation of SiteFinder, the other one being the plea for more time on the part of Verisign to notify whatever customers it may notify before rolling back the service. What kind of assurance do we have in terms of the future in terms of any possible reinstatement of the SiteFinder service that we will actually get some sort of advanced notice? And what sort of advanced notice do you propose to give us?

Chuck Gomes: it would be our intent to give good advanced notice in the future.

Robert Seastrom: can you quantify that?

Chuck Gomes: no, I don't think I can because I don't know the time frames in terms of what we're talking about right now. You know, the best I can do now is share my own personal opinion in terms of advanced notice.

Let me ask you a question instead. What kind of notice would you, in your business, your organization, need to be able to adjust?

Robert Seastrom: I think that that's more of a question for the community at large than for me in particular. I'm fairly technically savvy and rather agile. It took me less than 18 hours to adjust to the initial deployment of SiteFinder.

Chuck Gomes: and that's pretty much been our experience with regard to issues -- adjustments that people had to make -- make on their systems, whether it was with regard to a misconfiguration or whatever. It's been relatively quickly.

But one of the bits of information that will be very helpful for us in the process in the next few weeks would be to get that kind of information so we have a good understanding of that.

Robert Seastrom: thank you.

Steve Crocker: let me add one thought. One of the issues that has really fastened our attention, I think, is to make a very clean distinction between the core services that are at the base of the Internet, the core plumbing or, if you like, a different metaphor, the timbers that hold up the rest of the house, versus all of the many, many services around the edge and the innovation at the edges that has made the Internet such a vibrant and explosive and productive environment.

Where do you draw the line in terms of the sort of preservation of the stable core versus the innovation of the added services?

Chuck Gomes: first thing I'd say, Steve, is that we're absolutely, 100% committed to stabilizing the core com/net registry services. And, in fact, we have seen no degradation of that service at all, and we're responding, as most of you know, to over 10 billion queries a day on our system. So there has been no impact on that at all. And we're totally committed to maintain that kind of service, as we have done. And that's why we've made such huge investments in the com/net infrastructure over the past several years.

Ben Turner: and just to clarify, be specific for you, what we view is the basic registration of a domain name falls within that. And the operation of the name servers within standard compliance forms or formats falls within that.

Steve Crocker: one of the questions that has been circulating in the background here has been that with respect to the kind of guidelines that you're talking about for registries, that it would appear that if everybody followed those guidelines, it effectively takes the error response, RCODE 3, and deprecates that out of existence. Would you like to comment on that?

Ben Turner: I guess, you know -- are you asking if you follow the RFC standards around how to implement wildcards, is there an issue with that? I'm not quite sure what your question is.

Steve Crocker: well, are you taking the position that it's -- that that's sort of an appropriate or maintains the stability of the system to be able to make that shift throughout the top-level domains?

Ben Turner: we maintain that if you make the change in a standard compliant way, that it does maintain the stability. And I think the results bear that out. You know, we've resolved, before SiteFinder, over 10 billion queries a day with 100% reliability. We did that after SiteFinder, over 10 billion queries a day, with 100% reliability.

Steve Crocker: all right. We take one more question from the floor, and then it will be an appropriate time to move on.

Jonathan Zuck: I'm Jonathan Zuck.

It seems like the desire to kind of put it out there and try it might have been driven by missed opportunities in the past or something like that, because of the sluggishness of the vetting process.

And I'm just curious if there's been instances in which coming out and discussing things in forums has led to some sort of competitive disadvantage or some kind of a delay or missed opportunity that gave you the impulse to -- with testing and just try it to see if it would be as bad as the chicken little predictions would have been otherwise.

Ben Turner: first off, we did extensive internal and external testing before we did this. We did not do the wildcard implementation on an impulse. One of the things we're looking forward to is finding a forum we can look at, the community in ICANN to innovate. The track record hasn't been great, unfortunately.

A good example are multilingual names. Over the last two years, we've worked with the community to try to move that ball forward. And, basically, where we are is we've agreed to the guidelines that ICANN has established. We're compliant with the IETF standards, and still we can't roll out the service. So we have to find a way that we can address the need to work with the community and notify the community in advance but at the same time innovate.

Steve Crocker: okay. With that, I think it's appropriate to bring this session to a close and to move to the next speaker.

David Schairer is VP of software engineering at XO Communications and will be talking on the observed effects.

David Schairer: okay. I'm learning.

My name is David Schairer. I am the Vice President of Software Engineering for XO Communications out of our San Jose office.

I'd like to begin today starting talking about a few things that actually didn't break when SiteFinder was pushed. There's been a lot of chatter online, NANOG, Slashdot and so forth, about some things which were vulnerable which did not actually occur.

The first and foremost of this that one saw in the -- is that better?

-- in the chatter was that the new wildcard records could be used for denial of service attack. Specifically, that anyone could effectively stuff a caching server by using the near infinite number of records in the wildcards to stuff it and break it.

This is not the case for two reasons. Wildcards existed anywhere and any wildcard would have the same size of near infinity within it. And, two, since most DNS servers -- most caching DNS servers today are already doing NXDOMAIN caching of a TTL around the same size, around 900 seconds, this is not actually -- does not actually change the behavior. So that is a non-issue.

Secondly, although a lot of us have been sort of sloppy in the way we communicate this, there is no direct impact to DNS within already delegated names, only -- wildcard only impacts second-level domain names with com and net that do not exist.

And thirdly, although I have seen a lot of people saying that there are issues with reverse DNS, no one has really spelled out what those are, and, in fact, reverse DNS is a completely separate system and is not impacted.

Now, moving on to the bulk of the presentation, things that were impacted. Scott addressed some of these things in his talk. I did not have time to modify my slides. But I will respond to him where I can.

From a web perspective, the wildcard IP server that only handles port 80 requests, anything else besides SMTP, any other services that are used as part of the web at large function less appropriately than they did before. In other words, error messages that were previously presented for domain names not found are now presented with varying kinds of network errors, FTP, RTSP, any other public or private extensions to, you know, the browser relationship.

Likewise, there's no HTTPS, so the user will get a worse rather than better experience. The wildcard redirect, the one that handles the first web hit doesn't support or at least did not support, at the time it was taken down, HEAD and POST and some of the other standard HTTP components.

If you come in and type your name, the site you're looking for in a browser, you'll probably get the SiteFinder. If you have a bad link on your web page to a site that no longer exists, like a post or a link to a streaming media file or whatever, those things will not function. And so for the -- admittedly, the minority of web hits that are not HTTP get request port 80, these things actually decrease the response, the appropriateness of the response from a customer browser.

Lots of accessibility issues as well. In my testing, I tested with about a dozen different languages, both common and uncommon. None of them were -- did SiteFinder come up in accept language headers with my browser. They all returned in English. We all know that this is not an English-only Internet infrastructure. I think there are some statistics on it now that English will be in the minority within 10 years on the Internet. Obviously, there's no protocol requirement that web sites exist in multiple languages. Most web sites, the vast majority, exist in one language only. Infrastructure sites that try to cater to a multitude of different groups tend to be in multiple languages.

For example, I tested Google at the same time to verify my testing. I could not find the language that Mozilla supported that Google did not. There were a lot of languages that Mozilla didn't support, so I didn't try those.

This -- unfortunately, in most cases, people select their web sites, they self-select the web sites they travel to based on their language. So if the language isn't English, track English-reading users -- attract English-reading users, French attract French, and so forth. SiteFinder isn't one of the other sites out there where people don't have the choice to self-select.

Since there are multiple languages sites within com and net, there are many language users who happen to use the com and net infrastructure. But if they hit SiteFinder, they're only presented with English. Similar issues with accessibility and appropriateness for error handling. Especially browsers that are maybe readers for visually-impaired users, and, you know, alternate browsers in handheld devices and other nontraditional devices often have custom error handling for not found DNS responses that are appropriate for those devices. And those errors have been obliterated by this change.

Another thing a lot of people have been talking about is the network cost. Most of us have gotten pretty jaded by our high-speed, always-on Internet connections. But there are a lot of people who are still on tariffed or otherwise pay as you go connectivity. Whether it be ISDN, cell phone, satellite phone, from other parts of the world, whether it be RC 1149. Network connections that are -- where you pay by the byte.

These people are now getting roughly 100 times the amount of traffic down when they get -- when they go to a site that's not found because of the size of the -- the redirect and the SiteFinder response. It's more network sessions if they have a local cache, it would make no network connection at all. The size differentiation is entirely dependent on size in SiteFinder.

By Saturday, the SiteFinder web pages were extremely small and stripped down as far as the HTML size. If new data were to be added, for example, banner ads or additional information or additional content, this differential in size and network cost would increase.

Stability risks directly to the title of this forum. The root server infrastructure, as we all know, is designed to be multi-addressed and distributed as Scott said. It has extremely high uptime because of the way it's been designed. It's based on a lot of public peer review, a lot of open discussion about how that kind of DNS infrastructure should be done.

NXDOMAIN responses are no longer resolved purely by the root server structure, they're now going to a single address infrastructure that's seemingly centralized. It has not been peer-reviewed. We do not know the nature of the server farm. Scott said "server farm" several times. We do not know if it exists in multiple locations, how it has been constructed and built for security and stability.

And the customer experience depends directly on the response time to this infrastructure. If that server slows down, if it gets -- if it goes down, or I should say that server farm goes down, customer experience will be drastically worse in the previous NXDOMAIN response, customers will not know they are going to the wrong site -- they will think their site is down.

As Scott alluded to, this thing is likely to be attacked, both the wildcard IP and the SiteFinder servers it redirects to. We have seen, unfortunately, any number of distributed denial of service attacks and other, you know, no routing, other kinds of attacks on the Internet infrastructure of late. These things are only likely to increase.

And the SiteFinder service has basically made itself a prestige target that, by its publicity, by its very central placement in the Internet infrastructure, has made it a prestige target that will make it very likely to be attacked. And we need to understand fully the risks of what will occur when that happens.

Moving on from web to SMTP and e-mail systems, as Scott noted, there's a port 25 server on the Verisign wildcard IP. Initially, the server that was deployed is -- was badly non-compliant. It was replaced a few days later. The number of problems caused by the initial one, I'm sure, are legion, but it luckily was replaced reasonably rapidly.

The new server is protocol (inaudible) but it has issues that remain. Its response to an upload banner when you connect and when a client SMTP server tries to put a message to it, it limits it to 10 megabytes, any sender who is sending to an unknown domain a large message will receive back from their SMTP server a response that the message is too large rather than the proper response that the domain was not found, which would be a very confusing message for many people. And I don't look forward to having our support group diagnose that one.

Likewise, the client response -- the client timeout on the new wildcard SMTP server is 10 seconds on all messages. That is technically compliant with RFC 2128 -- RFC 2821, which says they should be five minutes. I did a survey. 10 seconds was shorter than anything else I could find. Some of them were in the sub-one minute range, but most within five minutes or more. This can impact slow senders on slow connections or congested connections and will cause mail going to unknown domains not to be bounced, but instead to be queued and retried at an unknown cycle, which will also impact user performance -- user reliability and experience.

I'm not prepared to talk to Scott's proposal of a wildcard MX. I didn't redo my slides as he was talking. Some issues will be changed by that if it's done; others will remain; and I think there will be others that are new. Scott did allude to the problems with MX and a interactions where an MX -- where there's an MX list pointing at multiple a records, one or more of which was previously unavailable and becomes available by the wildcard. He had said that this was a very small number of sites. I'm glad that the data mining was done, because I didn't have time to do it. He said it was about .07% or .077%, which is a very small percentage, but that still amounts to north of 15,000 individual domains that were impacted by this when the wild card was pushed three weeks ago.

There are two separate configurations where this breaks. Neither of them is, as Scott said, common, but these do both break. One is where a nonexistent a record is available only in a private DNS space and the secondary MX is public, so mail from the public domain networks normally route to the secondary, mail from the primary routes to the primary. We have definitely seen this and there was mail sent to SECSAC about that. When the wildcard was pushed, mail from the outside started to bounce.

Another similar case, but with a different cause, is when a customer -- when an end user domain has multiple a records in multiple domains and one of those domains expires. When the domain expires in the registry, it immediately gets taken over by the wildcard, and the wildcard then bounces the mail. Whereas previously the mail would have rolled to the secondary.

As with web traffic, the wildcard record leads to increased network cost, as well as server cost at mail farms. Previously, mail was bounced directly by the sending server, saw the domain didn't exist and bounce it had with only a DNS lookup. Now it has to do an SMTP session to the bounced server at the wildcard IP. That drives up network and server cost. If that server ever slows down, you'll have pretty extensive queuing at all mail farms.

If, for example, the wildcard IP had a denial of service attack going on for several days, you would have that mail queue up and you could see a chain reaction of other smaller mail systems backing up and slowing down that causes that chain reaction to impact the mail infrastructure at large.

Spam filtering. This has been the one, more than any other problem, spam filtering is the system that has gotten a lot of attention in the media. It did break a very common spam filtering rule. Scott said in their stats this was less than 3% of blocking. Internally at XO, I had originally reported 10% based on my first stats. I ran new stats yesterday and it was close to 20% for the data sample of yesterday. I don't know why spammers haven't caught on to this one yet. I'm not going to be the one to tell them. Of course, I just did.

(laughter.)

David Schairer: but I'm sure no spammers are ever reading this transcript.

So this -- our systems have alternate spam filtering that catches most of this as well. It's much heavier. It takes a lot more CPU time. A lot of sites that don't have the heavier, more analytical filtering, this will directly increase spam. The wildcard record also impacts spammers and legitimate mailing list operators in an interesting set of ways. Spammers may send a lot of mail to the wildcard and never clean up their lists. You can determine whether that's a good thing or a bad thing. The same thing can impact legitimate mailing list operators who may or may not clean up their lists based on bounces. But it certainly is going to change the paradigm there as well.

Lastly, Richard Smith alluded to a lot of this-- alluded to a lot of this in his question already to scott. Because the wildcard customer SMTP relay and misconfigure their e-mail to point at the wildcard, the error received by a customer in all MUAs, in mail software, not just Outlook, is extremely confusing. This causes a number of support and other difficulties.

We should move on.

DNS impact. A lot of alarms, a lot of ISPs and other service providers monitor for positive and negative responses from both authoritative and caching servers. This broke a lot of those alarms in a we're looking for obviously nonexistent domains. Set a lot of alarms off. Or in many cases suppressed a lot of alarms that may otherwise have gone off when a domain expires or when a domain or whatever reason fails to exist.

For example, the domain expiration is a big one. A lot of people have monitors on that so they see a domain if it falls out of one or more of their roots that valid response goes away, this problem occurs. We've talked about HTTP, we've talked about ISN’T. Scott provided interesting numbers in his presentation, which is one of the things I'm going to ask for about what kinds of traffic are hitting that wildcard IP.

As we saw, the vast majority of packets coming in were SMTP and HTTP, I would love to know if that's a Sim count. If it's a pact count, it will be misrepresent the session counts. However we can see from that very clearly there is a lot of data coming into the wildcard that is not handled appropriately by the wildcard IP. Some of that stuff could be handled appropriately, for example, one could -- an NMPT rejector that sends an appropriate error message back.

Some things like SSL can't be responded to properly. There would not be a valid certificate requested by the client browser. Much of that traffic will be protcols that are unknown to the general public, because they're private protcols, currently have existed only in -- only between known systems at whatever entity was running them, and now are hitting the wildcard server because of the wildcard IP.

These problems, we cannot predict universally how client software will respond to the change from an NXDOMAIN response to a response that points to a server that does not respond or returns a reset. That is based on each individual system. The problems that arise from this are in many cases slow to be found, slow to be diagnosed and may not cause immediate errors, may cause couping.

Someone is running an SMTP application, would use queueing. I've seen reports of printer problems, printers trying to contact unknown names trying to contact the wildcard server. Many of these things are embedded applications.

Firmware applications. Legacy applications that are not being maintained, may not have the level of sustainability. Required to replace them and repair them in a hurry.

Lastly, Richard Smith spoke to this.

Client configuration UIs, not just e-mail, but any client configuration that UI that checks for a valid DNS on a typo, mistyped name, instead of a useful error replaces witness a less useful error when the customer tries to use the system.

For most of us, like the speaker who said it took 18 hours to replace his infrastructure, this is an eye now answer for most of us, rather than a life-threatening or sanity-threatening situation. For nontechnical users often relying on remote phone support, where one not so cryptic error message is replaced by other much less useful error messages, this can be an extremely frustrating thing. It drives up support costs. As someone who worked extensively with support organizations doing support for software packages, it has required and will require ongoing retraining of a lot of support folks to watch for these errors and be able to train and correct configuration errors in customer sites. Many times remote customer sites, that have been bitten by the wildcard.

In conclusion, the wildcard deployment did not cause a single catastrophic pan network failure, the kind of thing that gets headline news, has media pundits talking about the hubris of the Internet and how it's going to collapse. Most importantly it failed to knock the California recall off of the headlines, despite my best hopes. What was caused was a large number of chronic problems, some of which are fixable with deployment, such as the SMTP protocol handling, possibly changes in the way MX is handled. Many of them are intrinsic to the concept that has been used here, and are unavoidable.

SSL handling on the wildcard IP, we're not going to be handle SSL errors effectively because of the way certificates are managed. Nor should you be able to handle SSL errors more effectively. That would violate the security intrinsic in what SSL does. Some of these problems have been cleaned up already by software patches and fixes.

Paul is going to talk about that next.

If the wildcard records are redeployed, many more things will remain to be fixed. It's going to become a reevaluation of old code, similar in concept, much smaller in scope, to the Y2K preparations which we had many years to prepare for, which is why it was such a minor thing when it actually hit. It's a similar kind of, go through all the code that looks at the network, and does, so the name resolution and see what broke, and go back and replace it. Beyond that, this becomes a sort of ongoing tax on the Internet and supporting applications on the Internet.

All of us who do sell Internet services will have increased support costs, will have increased network costs, more things to worry about. We have had to go and patch things that we need to maintain. It has become sort of a chronic background hum that we will need to worry about.

At the larger level, this has shown us, I think very clearly, how sensitive the Internet is to core infrastructure change. The Internet is, and the systems underneath it, are more complex and more ubiquitous combined than other deployed technologies. More complex and more well-connected, to different systems than the POTS network was. Certainly more connected than the power grid is.

Note that, this wildcard affected people all over the world. A major power failure was stopped and limited to a certain region because of the cellular nature of the grid in the U.S. DNS, as the most central part of Internet function at, the most centralized part, is the most sensitive to this kind of change. Everything ends up touching DNS, whereas routing can be done in a more isolated space that's hard to make a single change that breaks -- it's easy to make a single change that breaks everything when you make an error. It's hard to make a single policy change that breaks everything. The effects of the wildcard in com and net are much larger than any wildcard in other TLDs that have been done. If you look at the number of domains and the amount of traffic on dot com and dot net, it would be something that requires exponential notation to represent.

At the end of the day what we see is RFC compliance is the cornerstone on which we build our systems. But it alone is insufficient to judge the impact of change that happened. There's also an Internet-- the RFCs are statutory law.

But there is also a common law of best common practice that results in the way things are done, above and beyond the RFCs, that when it comes to core shared infrastructure if you are, these things need -- the best common practices in the way people function on the network and people's systems function on the network needs to be evaluated before shared infrastructure is changed, even if it's RFC compliant.

I'll turn it over to Steve for the questions.

Steve Crocker: thank you. Thank you very much, David. Any questions from the-- from cyberspace? On the phone? How about from the panel? Ken?

Ken Silva: I have a couple of questions.

You made a comment in your presentation, you don't know where the servers are or how many of them are or whether they're dispersed in multiple locations.

How do you know where the DNS servers are now?

David Schairer: the DNS, we know there’re multiple IPs, that gives you -- it's harder to black hole or steal the routes for all of them. That information, at least at the root server level, used to be a common knowledge. It's limited for security reasons but it's not only done by Verisign.

Ken Silva: you also commented that this is a prime target for denial of service attacks. So in your opinion, you don't believe that the existing gTLD servers are prime targets for denial of service attacks?

David Schairer: I do believe they're prime targets, they have been attacked. This is another one.

Ken Silva: in the summary, you mentioned that there were no catastrophic failures. Can you cite some examples of some technical failures which would be more than in worst case, an annoyance to a very small number of people?

David Schairer: I think that the errors that I mentioned already are considerably more than an annoyance to a small number of people. I think the spam fillering changes were, that a lot of people deployed the day it was done, were a tremendous annoyance. They, most people that do that, had increased the load and the cost on the system I think the fact that most service providers spent a week or so evaluating the DNS changes.

Ken Silva: what I think we're talking about are security and stability related issues, not usability issues, if you will, unless it creates a stability and security issue. I mean, evidently XO gets more spam than the spam software vendors, even estimate, that use nonexistent domains. But the fact remains that even, is a 10% increase in spam a security or stability related issue? Or is it an inconvenience to a small number have users?

David Schairer: I would say from a mathematical perspective, 10% of mail coming in being blocked with spam does not necessarily mean that if that were turned off it's 10% increase in spam. If 20% of mail coming in is spam and 10% is blocked by that rule, it's 100% increase of spam.

From a mathematical perspective, it's not 10% in spam, it's 10% increase in mail load. I guess the question is what counts as stability. Does a user who previously could access Internet services effectively, because they had reader software that was able to use, able to receive a domain error can't, is that stability when users are impacted in their ability to use the Internet. Or is stability only when the root servers themselves, their integrity, is diminished?

Scott said that, the root servers are not impacted by the change, it's still 100%. I do not argue with that. The question is, does stability impact-- does stability cover the layer four and five services that are on top of DNS. And certainly they are impacted by this.

Ken Silva: well, they're impacted -- I'm trying to work my way through the presentation. There was some sort of claims which appeared to be more usability related, and sort of personal preference related. But if a user has, I guess I'm trying to get to the bottom of this, if a user has a program or software which they've fat-fingered the wrong address in for whatever reason, it's not going to work in the first place.

Okay, do you agree?

David Schairer: I would agree.

Ken Silva: it's going to come back with some sort of response. Either the wildcard or NXDOMAIN response. Neither of which is the preferred response to the user, right?

David Schairer: um-hum.

Ken Silva: in that respect, what we're talking about here, is services that would never have reached their destination anyways, right?

David Schairer: well, that's correct.

I think stability includes enabling the appropriate response, not just saying that an erroneous response is any erroneous response. You could say it would be acceptable for stability, if not RFC compliant, for the root servers to drop requests for domains that don't exist. Drop the packets on the floor, let the DNS clients time-out. The customer would still get an error message. Wouldn't be the right error message, but it would still be an error message for something that was wrong to begin with.

The argument that any error message is okay when the customer or end user or any application is in error space, is not correct. Otherwise, why would we have different error code responses in the DNS protocols to begin with? Why would we have a difference between NXDOMAIN and surf fail, for example.

Ken Silva: that's it.

Steve Crocker: Russ?

Russ Mundy: David, you've given some good examples up there, in terms of the analysis.

Have you gotten to the point, again, similar question of the last speaker, of being able to provide input information that would allow us to identify things that are infrastructure in nature that, how they ought to be changed or how one can go about determining if a change in some manner will have, or not have, broad-reaching impacts down to the user level? How do we revolve the infrastructure and not run into problems like this?

David Schairer: well, that's a very good question.

I think a lot of my process in this, and might be the process of a lot of people when this first hit, for me was, engineers coming to me and saying, look what just happened, this is going to break six things. And us looking at it and saying this is going to break 12 things. Sure enough, the things that we saw that were going to cause problems were realized in the field. With our customers, e-mail into the SECSAC and other locations, Ari sent me a few, came in the SECSAC. A lot of the debugging of this, certainly for the people who are responding to it, could happen in theory, as well as looking at it in the field.

It's not hard to look at, you know, a server that doesn't respond in various ports, and know that the error response -- what the error responses are going to be for the customer. You can work through a lot of the changes because you know the protocols. At the higher level, how do you manage change, how do you manage change so these things don't happen? That's probably a policy question that I'm not prepared to address. But certainly it is certainly possible to analyze these things, in a broad basis before they happen. And see that they-- see what's going to happen when they do.

I could cite as a specific example the SMTP server that was originally deployed when wildcard went live, was woefully non-compliant. It didn't speak close to isn't unless you got it in the right order.

That, any public review would have taken, would have seen as problem and it would have been fixed before it was launched.

Russ Mundy: thank you.

Steve Crocker: any other questions from the panel -- Ray?

Ray Plzak: yes.

Twice now, I've heard, an issue that security and stability gets turned into a discussion of usability, and is that really -- it's not a security and stability issue. One of your earlier comments, you said, well, yes it could be. I would like you to go further in detail about how non-usability affects security and stability.

David Schairer: well, the word "stability" needs to be defined. I said this earlier. Does stability mean the root servers for com and net and root servers in general, and DNS infrastructure in general, returns the response we think it's going to return as compliant with the various RFCs for DNS? Is that stability? Returning data to a new location is outside the scope of the word "stability" in this context. Or does stability mean that DNS and the other applications that DNS enables continue to function in the way that they're supposed to function, and in the way -- so that users of the Internet and of the infrastructure that we in one way or the other support continue to be able to do so.

And I think if you asked an end user, if you asked someone who is visually impaired and uses a reader, if it's a SiteFinder thing that doesn't work with the reader properly, or a user that speaks only French and it's an error they can't read, whereas previously the browser could, I think in their mind, the mind of the common person, the Internet -- something is broken on the Internet, and that is a stability issue.

Steve Crocker: thank you.

Mark?

Mark Kosters: a quick question.

That is, there's been a number of assertions made, but it would be very interesting to see some quantifiable numbers to back these things up, so that we can actually assess the sort of scope, of this.

David Schairer: a lot of those numbers need to come out of Verisign. Verisign has the wildcard IP. The numbers that Scott presented I would love to see in depth. The analysis of that in full detail will answer a lot of questions I have.

Chuck Gomes: there's also the other side of this, of course, and that's from end users and so on that are out there, that are the ones that have seen SiteFinder and so on.

Mark Kosters: what sort of things have you XO as a company seen in terms of additional support calls and that sort of thing?

David Schairer: I don't have specific examples with me. I think certainly the things that Ari and Paul have challenged is a start. We have no centralized way to say how many people had Outlook break. There's no way to quantify that.

I talk about language issues. I would like to see at the HTTP level the language breakdown of browsers that hit SiteFinder. How many send language where EN and variants of it is not the first thing on the list, not on the list at all.

Those are stats only Verisign can provide. They can tell us a lot about some of these things.

Mark Kosters: again, it goes both ways.

David Schairer: agreed.

Mark Kosters: and it would be useful to see those things from your perspective.

David Schairer: I think that if there's a followup on this, I think we, not just XO, a lot of ISPs and service providers need to continue to provide that input.

Mark Kosters: great.

Steve Crocker: good.

I think it's time to move briefly to questions from the floor. Let me ask each of our panel members, committee members as well, as the people from the floor, to identify ourselves when we speak.

Myriam Sapiro: Myriam Sapiro from Summit Strategies International.

My question, David, is to patches. There's been widespread concern reported that the deployment of patches to counteract the impact of SiteFinder could add further instability to the system. And when I tested the system last week, it seemed like IE had employed patches successfully. I wonder if you can comment, please, on your view of both the impact of patches and the possibility of any counter-patch that could possibly be developed by Verisign, and what the impact of that would be on the DNS.

Steve Crocker: let me cut in. The next speaker is dead on that subject, and we'll spend a lot of time on it. Feel free, if you want to do a quick response.

David Schairer: there have been a lot of people with a lot of issues, a couple of things that we had to fix, not just in response to the wildcard, but the patch that is were out there, that were interesting. But Paul will speak to that in full detail. Any time there is a rush job on patching, we have more stability. As we've seen with how many cycles of patches there have to have been to fix security holes in client software.

Chuck Gomes: would it be appropriate to set the record straight on a few points made in that presentation?

Steve Crocker: I think, if there are factual things, go for it. And if it gets too lengthy, we'll find another place for it.

Chuck Gomes: It would be brief.

Matt Larson: Matt Larson from Verisign.

I would like to stress, I'm responding on the fly, since we didn't have a chance to see this presentation in advance. I'd like to echo Mark Kosters' remarks that, while I think this is valuable input, I think that additional specific quantifiable examples would be helpful to everyone. And I'd also like to echo, it really is not just Verisign. We only have part of the data. We need reports-- it's not enough to say “this class of things has been affected.” We need to have specific examples if we're going to get to the bottom and really assess the impact of this.

I'd also like to make a plea for terminology and precision. The root servers refer specifically to the root name servers. Anything below that is a top-level domain name service such as the dot com or net service which we refer to in Verisign as the detail server. I think that's very important, and that terminology was confounded at several points.

I guess, I'd also like to frame this very briefly in context, that as Scott mentioned, in his presentation, there are already many top-level domains, several top-level domains that have wildcards. To the extent that the presentation makes it look as if this is all of a sudden something new, and these effects were not seen before, that's not accurate in that context.

Very briefly to address some of the things you said, the HTTP server deployed, does support HEAD and POST.

David Schairer: I got 500 responses to when it had to do HEAD.

Matt Larson: from the engineers that we quickly exchanged.

Steve Crocker: I have the sense that this -- I deliberately let the clock run, this session was important. But is it going to-- let me promise that if we can take this up at the end, we can devote some time. This is the kind of thing that this time was reserved.

Matt Larson: okay. I would prefer if we could do it now. I feel like there were some issues raised, and they're going to be hanging there and we won't have a chance to respond while the issues are fresh in people's minds.

Steve Crocker: I don't want to get into the broader discussions about getting data and so forth. Those are too large for us to get at at the moment. If there are specific technical things, like the clarification on roots versus TLDs.

David Schairer: what is going to be the forum to get that broader data? It would be very helpful for us if we had a well established process to collect that data. So we can complete the analysis.

Steve Crocker: the essence of the Internet is that it's not a well established process. I can empathize that we'd all like a well established process then we wouldn't have an Internet.

Matt Larson: all right, let me proceed, then. And, again, I must stress that I'm reading my own handwriting which is difficult to read on a good day, let alone being taken quickly on a presentation that we didn't have a chance to see.

The SMTP reject server responds with a 550 response to receipt 2. I didn't understand the comment about size of messages being an issue, because one ought not to be sending a data command if one gets 550 response.

David Schairer: it's not the data serve. If you send a size you get a 1002.

Matt Larson: to the larger point of the SMTP server, one of the things mentioned, people's feedback on, is we believe a wildcard MX record pointing to a nonexistent target would be a much better solution to the whole thing.

We believe that that addresses all of the issues that you raised with regard to SMTP in your presentation. I guess I also -- you expressed frustration with regard to the treatment of the spam issue. I guess it's been frustrating to us as well that this one corner case has been, we believe, blown out of proportion. I think you somewhat alluded to that in the presentation. I'd like to point out that this check for nonexistence is one of many. And we just have a hard time with this idea that it had a huge impact. You don't see that to be the case. For example, the commercial providers that we've talked to have said that they don't even bother with this check any more; that it's not effective.

Steve Crocker: we really are into the kind of things that have to be thrashed out at length and policy clarification. There's a multi-step process that has been laid out and extended, in fact, to provide lots of room for clarification and so there will be ample opportunity before all of this is cast in concrete and print.

So I apologize. But we do have a requirement to get the rest of the material that has been prepared out. And we'll provide as much opportunity as we can ongoing here.

Matt Larson: Matt Larson.

Steve Crocker: So, Olaf, we need to cut this off.

Olafur Gudmundsson: very quick question, one?

Steve Crocker: no.

Olafur Gudmundsson: how much did this effort cost you?

David Schairer: I'll answer your question offline.

Steve Crocker: so we move on to Paul Vixie from ISC. Thank you all for cooperating with the process here.

Paul Vixie: okay.

As Steve said, I am Paul Vixie from ISC. If there are any other microphones on, please make the little red part go off so that we don't get the feedback. So I'm here to give a short overview of what I know of what was done by the community in response to the wildcard and the synthetic data that was being issued by the com and net servers for a week or two there.

First, because I think there might be some people in the room, and even on the panel, who are not aware of one of the basics of DNS, I will describe the three types of responses which a server can issue after it receives a query. It could give you -- if you ask a server for, let's say, the address that corresponds to a name, it could simply tell you, "here is the address that corresponds to that name." It could also send you a referral saying that it doesn't have the information that you want, but it knows of some other name servers that are below it in the DNS tree where you can presumably get that answer. Or it can tell you that the name that you want does not exist.

And the difference between those three types of responses is rather important to sort of understand in the community's response, since what Verisign had changed was that a number of things that used to come back as negative answers instead were coming back as normal answers.

Give you a little bit of my own background. There's a reason why I'm giving this presentation, and that is that we were the supplier of some of the work-around technology that was deployed during the time that Verisign was producing this synthetic data. We are a not-for-profit. We publish BIND. We run one of the root name servers. We do some other things. And our budget, you know, my salary, everything we do, depends on relevance to the community. And so when the community had the intensely negative reaction that it had to this synthetic data, we pretty much had to do something, and I don't want anyone to misinterpret our actions as having taken sides. I certainly have my own personal views.

But ISC as a company has only the view that the people who run our software ought to be supported. And, of course, we have no financial stake in the outcome of this one way or the other. The first workaround that hit the streets was a patch that was designed to look for answers containing the particular address that was being issued by the com and net servers in response to a query for a nonexistent name and translate that back to the negative response code, which we call here RCODE 3.

Now, the biggest weakness of this is that this address can change for perfectly natural causes. For example, if you were to install load balancing servers or try to put more servers in more places, there are a lot of reasons why this address is probably going to change over the years. And so hard-coding it into software is a bad idea. In fact, hard-coding it into a configuration file is also, I think, not the scalable approach.

And one enterprising person issued a patch for the BIND 8 software that had a complete list of all of the addresses being issued by all of the TLDs which currently have synthetic responses to nonexistent names, indicating that for one thing, people are willing to go with unscalable approaches; and for another, there are a lot of really enterprising people that thought that something really needed to be done not just about the com and net synthetic data, but about all synthetic data coming from TLD servers.

Another workaround, this is what we did, was to require that all servers from certain domains, and in this case, it would have been com and net, be of the type of referral or delegation, meaning that they would be neither positive nor negative -- actually, they could be negative or they could be a referral. And if they were positive, in other words, giving you an answer instead of telling you where to get your answer, then we would translate that into a negative response. And this, of course, does not depend on the specific address that is being returned and is, in my opinion, a more scalable solution. It is not the default behavior; it will never be the default behavior. You have to turn this on explicitly. You have to know it's there. You have to know how to use it before this will have any effect on you or your user community.

Very shortly after we released our initial patch, we were told of the long list of other TLDs that were also producing synthetic data. And I quickly realized we were not going to be able to enumerate the list of TLDs from which synthetic data was unwelcome because there was a potentially growing number of them. And instead, we reversed the sense of the new configuration directive so that you could enumerate the TLDs from which synthetic data was acceptable. For example, the .museum domain.

And this, if it became widely deployed, would have the effect of locking out future synthetic data from other TLD administrators who wanted to put that in. But, once again, this is not the default behavior. And by not default, I don't just mean you have to turn it on explicitly; I mean it's not in our example config file. So you really have to go digging for it before you'll find this.

People who can't run the name server software of my personal choice but have other versions of name server software that can selectively forward queries depending on the domain that that query falls in have been able to put up a server having this feature and then direct all of their queries for, in this case com and net, to that server. We have such a server that we run as a kind of a service to the IPv6 community that has this feature enabled. And although we have published the fact this server exists and has this functionality, the traffic on it has been very light, on the order of 20 or 30 queries a second.

So this is not really being used as a wide scale workaround, although the code for the -- the BIND9 patch has been downloaded about 30,000 times by about 15,000 users, indicating two files per user. And I know which two they were. A number of those were not end users, but were integrators, such as a Linux distributor or a BSD distributor. So the actual distribution of the patch is probably quite larger.

Another thing that I've seen done is an ISP has advertised this address to their customers. The way the Internet routing system works is that once you get behind the wall into a given ISP network, they have more or less complete control over the user experience behind that wall. So I have seen a couple of people put up ad servers at this address so that after Verisign was -- would supply the synthetic data and the user's browser would attempt to connect to a -- to a web server at that address, they would end up talking to a web server inside that ISP. This is sort of getting down the slippery slope toward instability, as all of these workarounds do, to one degree or another. This is one of the worst simply because the experience that you get depends on which ISP you're using. And it may be that Verisign will try this again at some point. And it may be that they would, you know, solve certain problems in certain ways, like certain types of mail servers or web servers or SSL servers, that people inside these ISPs won't do. So I'm hoping not to see this as a widespread thing, although we have to recognize the complete autonomy of an ISP to control what server is reached when this address is tried.

Here's the -- pretty much the far end of the spectrum in terms of how damaging and how -- how much instability you can generate with a workaround. This is where someone not only does the thing I just talked about, where they handle the local address or they remap the Verisign address into a local address; they also look for RCODE 3 responses coming from the outside and remap those to a local address also. And in this case, research shows that this change was made by a couple of very large, you know, multi-million-user access providers during the period after Verisign put their change in. So I think that it was directly inspired and it was in direct reaction to this change.

This is a short list of the software and the open source community that I was able to find that had patches either from the vendor or from the community or, in most cases, from both, since not everybody likes the vendor patches, these things. The complete list of these is in my bibliography.

Quick word about protocol violations. If you violate the protocol and you send a response that is not conformant to the protocol, then that response will be thrown away. So from a completely definitional perspective, Verisign is not violating the protocol. The responses they're sending are being consumed and the addresses they're sending are being contacted.

As far as that goes, none of the other workarounds that I have mentioned is a protocol violation, either, because all of those responses are being also consumed and the addresses that are being contained or the RCODE3s being contained are being acted upon.

Now, as to whether there's a higher-level problem that isn't so much a protocol violation as a -- I don't know -- a system expectation violation, that is not for me to say, except that, clearly, there is a lot of -- a lot of opinion on the street, heated in at least one direction on that topic.

This graph is a little hard to see.

Zitrin and Adleman have published a paper that's kind of on the same topic as this short talk and it, again, is in my bibliography. These slides, by the way, will be on the ICANN web site, so you don't have to take notes when you see those URLs go by later. The top two graphs show a couple of, I guess, notional ideas about what might have happened, which is either that the traffic to Verisign's SiteFinder site would have gone up and stayed at that level, or it would have gone up and then incrementally stepped down as various large ISPs disabled it or redirected it locally. And it's very hard to see on the scale, but on the bottom of this graph, what you'll see is that the workarounds haven't had a huge affect on the traffic toward Verisign, at least as measured by the Alexa user community. Another, it went up and stayed up. And there might be some hair in the small print at the upper right corner of that bottom graph, but it's hard to see. I believe that if I got an update of this graph today, it would show that the traffic had gone back down, because Verisign took out the synthetic data. So I'd love to see a study of that small-print area at the upper right.

MSN's distinct visitors, according to the Zitrin and Adleman paper, were down about 20 million a day. Verisign's traffic rank amongst Alexa users went from about number 1500 to about number 20. About 9% of the Alexa users that were submitting data during this period simply didn't see Verisign's synthetic data at all. We don't know how representative they were of the overall Internet population.

Adelphia was an interesting case, because every Alexa user connected through Adelphia did not see the SiteFinder data, the synthetic data for four days, but then started seeing it again. It may be they either got complaints or they -- I don't know what that reason was. I'd love to contact them and find out.

This one is really busy. It's really hard to see.

The two things that I hope to demonstrate with this, which, again, comes from the Zitrin and Adleman paper and is in the bibliography, is that something bad happened to SiteFinder on September 23rd. Don't know if they were retrenching; I don't know if it was an attack. I'm not sure what that was. But there wasn't a lot of traffic that day. But what's -- the other thing that's interesting is that this dashed line that is the -- what that is is all other networks, for comparison purposes. That has been dancing around in a very high range, whereas a collection of networks, most of which are in China, they were responding to SiteFinder quite a bit until after the 23rd, when they all started drifting down to zero. And by September 27th, this collection of a dozen or so Chinese access networks were no longer accessing SiteFinder. And I believe that the indications are -- and this is also in the Adleman and Zitrin paper -- indications are that china didn't like SiteFinder very well or they had their own local version that they wanted to run instead.

In summary, we've seen workarounds to try to turn back the clock. These slides were prepared during the time that the synthetic data was still coming out.
And it has been suspended since then. Seen workarounds to try to keep the revenue from this type of typographical error local. We've seen workarounds inspired by Verisign's SiteFinder but which are even more ambitious. And, from my perspective as a protocol and software person, the total result of this is incoherence and growing incoherence. The people who are responding to this are responding by making DNS responses less coherent than they already were. And that's not a direction I'd like to see us go in.

So I think that the total result in terms of DNS incoherence is that we've seen some instability. And there will be more if the service is turned back on.

This is my bibliography. And I'm taking questions.

Steve Crocker: thank you very much, Paul.

Any questions from committee members on the telephone? Any questions from committee members here? Steven Bellovin.

Steven Bellovin: Paul, of all of the workarounds that you have mentioned, the one that bothers me the most is, basically, ISPs hijacking a piece of the address space. It seems it was a very long way towards very undesirable outcomes and we don't agree on address space or names space, the reason I have proposed the so-called alternate roots. Do you have any comments?

Paul Vixie: I am interested that you are more disturbed by that than by remapping RCODE3 to a local address. But just the same, I agree with you. And the -- the ability to remap these addresses and to handle certain well-known addresses locally could really turn into a big mess. You could do that with root name servers, with the Google web page, as I think China did a couple months ago when they were offended by some content. And that is not the kind of thing that either serves the community, in my opinion, or makes it possible to further improve the Internet technology, since there are sort of band-aids already attached to it.

Steven Bellovin: one of the fundamental principles is that names should not be ambiguous. People add address, we can see the problems that can cause, I think this is another instance of it, which is why -- I want an address or name to mean the same thing regardless of who your ISP is. And --

Paul Vixie: DNS is intended to be a universal naming system. It's supposed to be coherent, autonomous, reliable, distributed. And pretty much every product or service that I have seen in the DNS space in the last couple of years has attacked one or more of those principles. And the result cannot be good.

Steve Crocker: any other comments from the committee? Any comments from the floor?

Harold Feld: hi. Harold Feld.

And first I'd like to ask if -- is it possible for you to talk a little closer into the microphone? I know I've been having a great deal of difficulty hearing you.

That said, did you observe in the patches as they've been deployed that some of the ISPs or other independent operators have been extending wildcard-like services to TLDs which have chosen not to deploy wildcards or in fact have taken anti-wildcard positions like dot org?

Paul Vixie: I -- not directly. But in one of the workarounds that I presented, what we're seeing is that a query goes out from some customer of some access provider, and it reaches a server, like, for example, the dot org servers, and it's looking for some typographical error that ends in dot org. And the dot org servers, which don't have synthetic data and don't produce synthetic data, don't have wildcards, so those servers send back an RCODE3 response. That response, when it crosses back into the access provider from which the query originated, the access provider is then changing that on the fly into a positive response for a local ad server.

Harold Feld: from a user perspective, is this any different from, say, the feature in the Internet Explorer browser which returns a -- which sends you to a search page?

Paul Vixie: depends on what you mean by "user." If by "user" you just mean somebody -- a nontechnical user whose only Internet application is a web browser, there's probably not a big difference.

But if you mean an Internet citizen who is expecting protocols to work the way that they're kind of -- have always worked, then the fact that other applications besides their web browser are also going to get this response and behave chaotically is a big issue.

Steve Crocker: Paul, just to follow up, even for ordinary users, are they -- if their client software does the same thing, do they have an option to change that or are they kind of stuck with that?

Paul Vixie: in the case of Internet Explorer, I know there are workarounds so that if you don't like the service that you get from MSN when you enter a typographical error, you can go in and edit the registry and turn that off. If, on the other hand, your provider is doing something to your DNS responses before you see them, then there's really nothing you can do to your browser configuration to work around that.

Steve Crocker: so, but editing a registry is not within the usual scope of what most users know how to do.

Paul Vixie: that's true. And other browsers give you this as an option rather than making you go edit a registry. And I know there are plenty of how-to pages and whatnot. But the fact is, there is a switch and you can turn it off. It's just hard to find the switch, as opposed to there just is no switch.

Steve Crocker: any questions from the floor?

Chuck Gomes: Paul, if I understood you correctly, in your fix, the one that you designed, there is an option for exceptions; right? And I think you mentioned some of the exceptions that were made. How to determine when an exception is appropriate and when it is not, and is that a user configurable thing or is that in the patch itself?

Paul Vixie: I know how to determine it for myself and for my own servers, which is to say .museum -- it doesn't have any delegations, as far as I know. And it happens to have a wildcard. But the point is .museum would be invisible if you didn't make an exception of it. The patch simply adds the ability to list your own exceptions. And there is no standardized list other than the example on our web site.

Chuck Gomes: so each user would have the ability to make their own exceptions in the patch?

Paul Vixie: absolutely. That's what we believe ought to be done, is that every name server administrator ought to make this choice completely on their own.

Chuck Gomes: thank you.

Steve Crocker: any other questions? With that, I think we'll move on to the next speaker. Thank you very much, Paul.

Richard, you have the unenviable position of being between us and lunch.

(laughter.)

Richard Smith: first of all, I want to say, my name is Richard Smith. I'm an Internet consultant based out of the Boston area, known mostly as a privacy person.

Today I'm talking a little bit more about, I would say, maybe surveillance, which is sort of the other side of privacy. I want to say thank you to Steve for inviting me down, as well as the rest of the committee.

What I was going to be talking about today was looking at information flows in and out of the SiteFinder site. This is an issue that hasn't gotten a lot of attention but I think is an interesting one here of basically trying to decide what kind of information is Verisign getting by running this SiteFinder service.

My introduction to SiteFinder was I was looking -- doing some research on these little ATM machines that are showing up in all the superintendent markets and stores. I'm curious about how easily they are to hack. So I was in Google looking around at a particular model, which is the Mini-Bank 1000. And I clicked on a link at Google which the domain name had expired. And so the -- what we see down there in the second URL is the URL that the -- was received by the SiteFinder site. So you can see in that URL they get the entire original URL that I had typed in as well as the host that I was trying to go to. And, in addition, of course, as part of the HTTP protocol, they're going to be getting the referring URL from Google, the search engine.

What's interesting is to compare that to what happens by default in Internet Explorer. If you mistype in a domain name at the top of the screen in the address bar, the only thing that Microsoft gets is actually the domain name in question. It doesn't get the referring URL. There really isn't one because you're typing in things. And it doesn't get the whole URL that you were dealing with. This is one of the few times I can sort of point to good behavior by Microsoft when it comes to sort of data collection practices.

Now, what was interesting is, when I got to the SiteFinder site, I run a program called Bugnosis, which monitors third-party monitoring of web sites. And what I ran across here is what's called a web bug. It's basically an image that's embedded in a web page. It sends back information from a browser back to a third party of some sort or other. And they're used a lot of times in Internet marketing and advertising to measure effectiveness of advertising, things like this.

In the case of SiteFinder, Verisign has hired a company called Omniture, which is basically a web monitoring or web bugging company. They're in the business of trying to learn about visitors who come to web sites. And I find that kind of interesting that they would have this kind of service on something for mistyped domain names.

What you can see in here, this is a URL of the web bug. It's generated -- the web bug itself is generated by some Javascript code, about 50 lines of Javascript code embedded in the SiteFinder page. You can see, in addition to, you know, there's some things like what my screen resolution is, you can see why a web site might care about that.

But in red here, the original URL that I had typed in is being sent in to Omniture as well as the referring URL from Google. There's a lot of information being shared with this analysis company, who is Omniture. This is how they describe themselves.

This is the president of the company talking. It says imagine a device that could be placed in the front door of a department store to tell the store manager all kinds of detailed information about customers, what store they came from, who they were referred by, if they had been to the store previously, et cetera. You know, the usual sort of marketing hype around what information can be gleaned from visitors to a web site. It's basically looking at that information that's in the web bug to try to get this stuff. It makes it sound like you get the e-mail address of people. But, in general, that's not true. It's just the sorts of information that's embedded in URLs.

To give you an idea here of some of the customers of Omniture, Alta Vista has a really nice scheme where you can find out web pages that have web bugs on them. There's a million pages that have Omniture web bugs. They work for many different companies here. You know, all very name-brand companies here. We have Macromedia, Centers for Disease Control Atlanta, Oracle, NFL, the Church of Jesus Christ, the Latter Day Saints, the Mormons. Everybody wants to know what's going on at their web sites, just as an example.

Now, earlier, where this starts getting a little more interesting, one of the earlier speakers mentioned he thought the POST didn't work. That's not been my experience. I found the POST worked just fine. And that certainly makes the situation more interesting.

I have a sort of a fake page here that I threw together which is kind of like an e-mail page where you have “to” and “from” and a little message. And this post -- it does a post to a URL where the domain name has been mistyped. So the idea here is that you would have, say, a sign-up page or some kind of web form that would send data to a third party for processing. You might have, like, a Perl script that would send this off as an e-mail message.

So the question is, well, what happens in this situation? And the answer is, that information, then, is going to go into the SiteFinder site, even though this is not the intention of SiteFinder. SiteFinder is all about mistyped domain names. But here we've got web data that's going in. So the request here that we have, we had the POST HTTP request. If that's going into the -- the browser thinks it's being mis- -- the domain's -- some domain that's not Verisign, but it actually is. And then the actual POST data. So the receiving communication -- they're receiving communications in this particular case of an e-mail, which is different than their SMTP server, which only gets the to and from e-mail address. When SiteFinder receives this post request that it redirects from the mistyped domain name to the SiteFinder site at Verisign.

You'll notice it's a little bit hard to see in here, but I highlighted the word "POST" so it was green in the URL. For some reason, which I'm not sure why, but Verisign considers it important to somehow measure the amount of POSTing data that they're getting here. So I think they're aware of maybe misdirected data.

Steve asked not to talk about legal stuff, but I'm just going to throw out this one slide quickly. This reason why this might matter is that when you have data coming into a web server for somebody who's pretending to be somebody else, it begins to get interesting -- it starts raising interesting legal questions with something called the Electronic Communications Privacy Act, which deals basically with wiretapping. And I'm not a lawyer; I'm a programmer. But -- and I haven't totally understood this law here. But it is certainly going to be an interesting thing for a lawyer to look at here of how this POST interacts with this particular law here.

Another example we have here, this is a common thing, again, I just dummied up a web page. You're going to subscriber to a newsletter. If you pass it off to a third party and they've gone out of business or whatever, that data is going to be going into -- you know, be going into the SiteFinder site. Now, what's interesting is, the -- so what happens is, that e-mail address, because it's a git form, gets put into the URL. And then that URL is passed off the SiteFinder site, who then passes it off to Omniture, which is our company who's doing -- watching everybody walk in. So they're getting in, you know, this e-mail address in here passed off to this company in Utah who's sort of watching people as they go around the Internet.

It's interesting, you know, like on the SMP side of things, Verisign makes a representation, "we have no interest in this data." but in this case here, that data is being processed by a company who is in the business of sort of watching people do things. This is called, classically, sort of the technical term is a data spill, where you take git form data and pass it off to other people who really probably shouldn't be seeing it.

But it's an example here of when you put these systems together sometimes you have to be very careful of how you move the data in and about. The other thing is, to look at here, I think it's important, is what are the different kinds of HTTP requests that we can be dealing with on a web page?

The SiteFinder service really is to address the first two, which is you have a mistyped URL at the web address line, and then -- or if you click on a link, say, at Google, to an expired domain, you get to the SiteFinder site. That's the goal of the SiteFinder system. And as I already mentioned, though, that there's other sort of side effects or collateral damage of putting this system in place where the SiteFinder site has to handle these other kinds of HTTP requests because it can't really distinguish between the two. One is the form submits, where the receiving data in -- they're receiving data in for forms of expired domain names.

What else is interesting and I haven't heard a lot of talk about is you can have HTML elements on a web page that go to expired domains. So it might be like a picture. That's going to go in the SiteFinder site and it's going to respond with, you know, for an image file, what's supposed to be an image file, it's going to return, actually, the home page for SiteFinder. It'll do the same thing for script files and for frames. So we're sending back kind of like garbage data to the web browsers. Most browsers can handle all this stuff, except you may get a funny arrow if you have a script file in a web page. Suddenly you are returning what's supposed to be Java code, you're returning HTML.

There are some oddities. This isn't really a fixable problem because the SiteFinder site really doesn't know what kind of HTTP request is being done. When you do a HTTP request, you get a lot of things, IP addresses of the browser, URL with an expired or mistyped domain, the referring URL, cookie value of an expired domain. All the stuff is being sent in. Probably a lot of cases, nobody cares, but that's going in there. You can have hidden directories in a web site that now -- you know, that nobody's really supposed to know about, but now that information is being sent into the SiteFinder site. There's a lot of sort of data that's going in there, into the site, more than one would expect.

Now, one other quick thing here, I think there are alternatives here. I just want to bring this one up in sort of closing. If you really look at SiteFinder, I think the fundamental issue here is, it's trying to address a certain issue, which is to provide meaningful error message when you do either a mistyped domain name or you click on an expired domain name, you know, which I think is a fine idea.

The issue is that there's a lot of other places this affects beyond the intended target. So one of my questions is, why not have SiteFinder as a -- a little applet that runs in web browsers and provide this service, you know, to users who want it?

You know, just as an example of such an applet here, this came out a couple weeks ago here. It's called a Q Host Trojan. What they did is they did hijacking of the DNS servers at the client side here. And it has the advantage of -- by doing a client side, you can do a much better job of redirecting for when you need it, mistyped domain names and expired domain names, and you don't have to, you know, mess up things like people's e-mail software and so on.

Thank you very much.

Steve Crocker: thank you. Questions from members on the phone? Panel members here? This is going to be quick. Anybody from the floor? Uh-oh groundswell. Please identify yourself.

Jonathan Zuck: yes. Hello again. My name is Jonathan Zuck.

I think this is a really interesting presentation -- oh, the Association for Competitive Technology. We're an IT Trade Association. I am a software developer. And I feel like this is a really interesting presentation, but it falls into a trap that happens a lot here in Washington, which is separating intentions from technology and things. So I look at it as an example of like a form submit that's to an incorrect URL. And that seems to me like such a small percentage chance of happening because almost all of those are because you have gone to some sign-up form inside of a web site that's posting back to its own domain that you've already come to. So, yeah, some deep link that you've saved or something to a form might have that. But the percentage that some of these things would happen is very, very low.

And then the second part of my question for you, again, is, you know, Verisign's in a position to collect this kind of information using some other technology than SiteFinder if that was their intention to do that. If they're in a position to collect referral to see these URLs, to see hidden directories. If it's their intention, they could do it. Internet Explorer you said doesn't have access to referrers. They know the URL you were sitting on before you made the next request. They have access to referrer information as well. Everybody throughout the Internet have access to information. It's a separate policy question whether they collect it or whether it's used. I don't think it should necessarily be used as a reason to slow down innovation in the platform itself.

Richard Smith: those are good points. We get down to the data that's in there. What becomes interesting is the fact that you have Verisign pretending to be someone else in the case of an expired domain name. I think that does raise legal issues that need to be answered. And whether -- what their intentions are, it's hard to say. But we've also heard from Verisign what we have no use -- any intention with this. We see data being passed off to Omniture with no explanation of what's going on.

Steve Crocker: so you've said the magic word about legal uses. So I need to sort of pull that aside from sort of on the spot there.

Next.

Andre Carter: Andre Carter. He stole the first half of my question. The second half is you refer to SiteFinder as an error message. But in talking to folks and I consult with a lot of small technology companies, they kind of see it as a service for end users, that they can then do something with. And I guess if you think of it as a service to end users, I guess, rather than citizens, I'm not sure about that distinction, for a second, then maybe the collection of that data is something that can be used to make the service better to those users.

I listened with interest to what you were saying. And it came from one kind of perspective. And I wanted to kind of hear the end user side of that. If they are going to be able to -- yeah.

Richard Smith: on the end user side, I don't want to disagree with the idea that when you get an error -- you've typed in a mistyped domain that you may want to look at alternative spelling, that sort of thing. I think that can be a useful thing.

There are alternative ways to implement that. I think on the client side is an excellent place to do it because then you don't have any of these other questions come up about, you know, Outlook now giving you the wrong error message or an FTP software giving you the wrong error message when you're trying to use things. I don't have any problem with the end user side of things. But let's do it the right way.

Steve Crocker: go ahead.

Ben Turner: my name is Ben Turner. So help me understand how Verisign is hiding who they are on this site.

Richard Smith: I didn't say that they're hiding themselves on the site.

I don't know where -- I'm not sure what you're referring to.

Ben Turner: I didn't understand your prior statement, then, where it's not clear who Verisign is when the user gets to this site. We don't -- you know, Verisign doesn't disclose who they are.

Richard Smith: I don't think I made that point at all.

Steve Crocker: I think what you were saying is that Verisign is acting -- is returning a response when they're being asked --

Richard Smith: oh, that. Oh, yes. Oh, absolutely. Yes. In that case, that is a little bit different. What I was saying is that in this case here, if you got an expired domain, you know, fredscars, let's say, is expired and Verisign is responding that they're fredscars. And part of that protocol is saying, "I'm fredscars" when you're doing the POST there.

Ben Turner: so help me better understand that issue. The concern is that when someone types in "fredscars" because it's an expired domain name and the user gets the SiteFinder page back, the user thinks that's fredscars page?

Richard Smith: no. The software was sending in data to what it thought was fredscars when it's actually Verisign.

Steve Crocker: Ken.

Ken Silva: yeah. I think what Mr. Turner was referring to earlier was -- and you may not have even realized that you exactly said it. But what you said was that Verisign was pretending to be somebody else. And that is not the case. They may be responding for a web page that was meant to go to a place, but they're not pretending to be fredscars; they're not pretending to be Yahoo! or anyone else. I just want to be clear about that because that's definitely not the case.

Richard Smith: well, at the HTTP level right here, it is.

Here's the example. It says the host here is -- is some long domain, like fredscars, except this POST is going at this point in time is going to Verisign, before the user has any idea who they're talking to. It's sort of a legal issue, a legal splitting the hairs issue. But it clearly is Verisign's pretending at this point to be someone else.

Ken Silva: doesn't it say "verisign.com" at the end?

Richard Smith: I wrote that in there. That's what I chose to call the domain.

Steve Crocker: we need to tail that off.

Next.

Huey Callison: Huey Callison. I'm not here representing anyone. Since your preparation seems to be focused on the privacy, could you please go into the stub mail server they had running on that IP? When I tested it, I noticed it wasn't very RFC-compliant at all. It would return 200-series responses for the first two pieces of data for it and then a 550 for the third one. But having been a mail server administrator in the past and being in a position to read other people's e-mail, I have to admit that most of it is incredibly tedious and boring stuff. But having all of the MXs for every time someone makes a typo send an e-mail to Verisign, Verisign now gets to read every typo e-mail. Could you address the privacy implications of that.

Richard Smith: I didn't bring the SMTP server in because I thought it was going to be addressed more to begin with. But the key thing to remember with the stub e-mail -- SMTP server, it only receives two pieces of information before it aborts the transfer, which is the -- the receiving e-mail address and the "from" e-mail address, the envelope. It doesn't receive the content of the message. Or it doesn't -- it doesn't say that it's an SMTP server, so it doesn't try to get a password if you have to have a log-in for eSMTP. But it does get these two e-mail addresses. What Verisign says is, "we throw these away. We have no intention of every using them."

Huey Callison: But what's to prevent them from actually putting a real mail server there and trapping it and getting the message?

Richard Smith: nothing.

Steve Crocker: so in the interest of sort of being precise about this, there's observable and unobservable parts to this picture; right? So as you said, today, the "from" address and the "to" address go into their system. That's observable. The body of the message does not go into their system. That's also observable from the outside.

So what's not clear to the external observer and has to be taken as a matter of trust or possibly some sort of inspection is that the data that goes into their system, the from and to addresses, is disposed of, not kept, not used, and so forth. But if they made the change where they looked at the body of the message, that would be a different observable event; right?

Richard Smith: right. I mean, you can just run a packet sniffer and you can look at what kind of transactions they're doing there.

Steve Crocker: I think it would be more obvious than that; right? Because the sending server would observe that the body of the message moved, whereas now the sending server knows that it doesn't.

Richard Smith: I don't run e-mail servers. So, yeah, that sounds good.

Steve Crocker: yeah. All right. Even I know that. And I'm not much of a mail administrator.

All right. Any other questions? I saw you twitching there, Ken. Did we cover the aspect properly?

Steven Bellovin: actually, Steve, this is Steven Bellovin again.

As far as mail servers are concerned, it's not completely clear what all mail clients will do if they send a -- they might get back some -- well, they sent the recipient two lines and get back some yes, some don't exist, they may or may not send it. In the case where you don't get back any valid recipients, you clearly shouldn't send, I'm not prepared to say no mail clients are that stupid. You agree that the mail client whoever wrote it would know. I certainly don't have an exhaustive survey from them. Maybe someone from Verisign can answer that.

John, do you know? You're a mail expert.

(inaudible.)

Matt Larson: actually, I think I should defer to the author of RFC 2821.

(laughter.)

John Klensin: John Klensin. The answer to your question is that if several receipt commands are sent, for those of who you are not mail system experts, what that means is more than one recipient on the message, the first issue is that the client cannot give up after sending one of them because there may be more. So they keep sending receipt commands, and a smart client, if it sends receipt commands, it gets no answers to all of them, will stop and then not send data. But there are known to be a number of clients, at least one of them widely deployed, which, given multiple recipients, will not notice that it has gotten negative responses and will send the data, responding only when it -- telling the user there's a problem only when it gets back a negative to the data being sent.

Steve Crocker: oops.

Matt Larson: just a brief comment to echo something I said earlier and to make a request for information that was in Scott's -- I believe was in Scott's presentation, that one of the things we are considering, should the service be relaunched, is a wildcard MX record with a nonexistent target, which would, we believe, eliminate all of these issues. We would not run the SMTP server. We would be in no position to even collect this information even inadvertently.

Steven Bellovin: I personally think that would be a very good change to make.

Thank you.

John Klensin: it won't work.

(laughter.)

Steve Crocker: gosh, that's a great sort of pregnant pause there.

And I think with that, we will take the break for lunch right now. We have run a few minutes over, but, to my astonishment, not nearly as much as I feared. 2:00 sharp we start up here.

(lunch break)

Steve Crocker: let's get started, folks. Let me ask down. Make a decision whether you're in or out. It's okay if you want to use the hallway. I guess this is the place where I say there's plenty of seats down front, if you want to move forward. Everybody is comfortable where they are.

Steve Bellovin, AT&T research, is our next speaker. I've known Steve quite a few years, he is an incredibly smart guy.

What's making all that noise?

Steven Bellovin: we'll try to sort it out.

Steve Crocker: take it away.

Steven Bellovin: is the projector on? There we go.

I'm going to be talking about some of the protocol and technical issues that underlie some of the -- architectural issue that is underlie why a service like this is problematic. Not so much the motion of typographical, that's a fine service. Doing it where it's being done, at the very center of the net in some sense, is inherently going to cause some problems.

A lot of the problems stem from what the DNS is, versus what is isn't. In particular, there's a big confusion between the notion of a directory service and a lookup service. What's being implemented here is a directory service, which provides an imprecise, often interactive service. Directories are great. I wish we had more of them. Search engines, in a sense, are the nearest thing we have to directories on the net today.

The DNS was not built to be a directory service. It was built to be a lookup engine. You give it a query string, you get back a reproducible, predictable answer. Subject to rigidly-defined area of doubt and uncertainty, change of certain names, you always are supposed to get back the same answer too a human being, often what you want is a directory service.

A lookup. I don't remember if Verisign is spelled "sign" or "sine." I'm a math geek. But that's great, if I'm doing some interactively. It's not good if I have a program that's trying to get the same answer back all the time. And I guess the best example of it is do I get back the exact page I'm looking for, when I get back an unambiguous answer to the query, or do I get interact some more, when you get an ambiguous result.

The DNS was designed to be the latter. There are many other programs that rely on this behavior. As we get surprising behavior. Programs no longer do what you expect. Sometimes it's incorrect behavior. Sometimes the error is presenting itself differently.

We heard a lot this morning about, for example, how to configure your mail client so that you don't get an odd ball error message, you typed in the wrong name for the SMTP server. It's not that the message is necessarily wrong, it's that it was different, it was not designed to handle that. From an architectural perspective, preemption is a more serious issue.

Philosophically the thing that made the Internet so powerful was that it empowered the end points, not the middle. If you look at the older networks, the phone network in particular, the intelligence is all in the middle of the network. The Internet is designed to follow the smart-edge, dumb-center model. And you want to make sure that the center does as little as possible so that it does not prevent the end from doing the right thing in their particular context.

Wildcards at this level prevent this ability and put control in the center. So to see exactly what this caused, wildcards work more or less adequately for most web work.
Yes, there are exceptions. Disabled some existing typo-correction mechanisms, causes problems for people who don't want to get English language search pages instead of their native language error page. Things like that, you might be able to work around, you might not. It worked well enough most of the time for ordinary e-mail, once the original egregious implementation of SMTP was fixed. That was done quickly.

Wildcards are not well tested, not well used except for e-mail. This was, in fact, noted in the IAB's response much the primary use in reality for wildcards has been for e-mail. So we need to say, what is the minimum common protocol set for the Internet.

If you look at-- historically the Internet has an modeled as an hour glass. Model over transport processes, run IP over more or less arbitrary link layer device. Actually I've been joking-- IP over carrier pigeon, IP over signal drums. Generally we prefer things like Ethernet and Wi-Fi and so on. But IP is in the middle.

The question is, is the new model everything over HTTP, the web protocol. If you look at today's picture, IP, again, is sitting in the middle, that is the common substrate that everything on the Internet is supposed to run over. We lay our applications on top of transport protocols, usually TCP or UDP, though for newer ones that are starting or important. But fundamentally, it's IP. That's in the middle. If you want to build something new, you layer it on top of IP, or on top of a transport protocol. If we make new services web centric, HTTP centric, we've moved things up. Applications have to be layered on top of HTTP in order to get universal connectivity.

To some extent that's a trend that's happening for lots of bad reasons to do with firewalls. We have lots of things that are layered on top of HTTP that shouldn't be. We don't want to make matters worse. We especially don't want to make matters worse in ways -- places where you can't opt out. And the reason is fundamentally innovation. Most of the interesting new services come from the edges.

The IETF, I'm more than security area director of the IETF, working with it for ten years or so, does not invent most new services. The ISPs in the middle of the net don't invent most new services. The web, is the most glaring example, invented off in this, to many people, obscure physics research lab in Switzerland. But it had lots of merit, had lots of usability. And fundamentally, you did not need cooperation from the ISPs or from the IETF to deploy this new service.

More recently we've seen peer-to-peer, as a major factor on the network. Peer-to-peer is often used improper and/or illegal purposes. But that isn't the point. It's a new service that did not come from the center. It came because some college student had a bright idea. And was able to deploy it.

I don't know what the next killer application is going to be on the Internet. If I did, excuse me I have a meeting right now with a VC.

(laughter.)

I do feel confident it's not going to come from the IETF, the IETF standardizes things until someone comes in with more or less a rough sketch. It's probably not going to come from an ISP. Even if it does come from any one ISP, it won't be a lot of good unless it's coming from -- used by people who are lots of customers of ISPs. No one ISP controls that much of the network. It's going to come from some place out on the edge. It will have to be usable by people on the edge, without the cooperation of the center of the net.

One of the jobs, or the center, therefore, whether it's the ISPs or the people running the essential DNS infrastructure, is to stay out of the way of each other's innovation. We don't want to lock the Internet into the mostly-HTTP model. That is going to stifle innovation, and from a security perspective, is going to cause security problem, because it's very hard to selectively filter on HTTP opposed to "let it in" or "don't let it in" through the firewall.

Wildcards break things because the DNS doesn't know what the user is doing. The purpose of the DNS is to take a host name, give back an IP address. The DNS does not know at any level that the user wants to get to HTTP or e-mail or some other new service.

Now, you might argue that's an ancient mistake made when the DNS was designed close to 20 years ago. I will claim it's not a mistake. If you look at IP, IP doesn't know about services, either. The service identification is above, it's carried in the transport protocol. All IP knows about is IP addresses. Therefore the DNS gets back an IP address.

I've heard people argue that the service should be part of IP and not TCP. It's an interesting philosophical point. We can go back close to 25 years, and revisit that question. But it's fairly strong sense too late to do that for the current Internet.

The DNS, the semantics of what the DNS returns are quite closely matched to the semantics of what IP, the center of our hour glass model, wants. For that reason, the DNS can't ask for services. It is architecturally improper. It doesn't mean we don't want a directory service that knows about services and more. It does mean that the DNS is not the right spot to put it.

A consequence of that, a service-- if we have a wildcard optimize the behavior of the of the servers for one purpose, for one service, it will often work poorly or not at all for other services. We have heard claims that e-mail doesn't always work right. It certainly is the case that trying to black hole the routes of the servers goes very bad behavior in the middle of the network. That certainly is a change. I'm not advocating people black hole routes, but causing that change.

I don't know, for every service that's in use in today's Internet, let alone the ones that haven't been invented or are in someone's garage, I don't know what the outcome will be of getting a "connection refused request the "message or "con investigation" rather than "no such host." I don't know what the fallbacks are.

We had a discussion at lunch about John Klensin's comment about having an incorrect, nonexistent host name. John said that wasn't going to work. Exactly why not? I don't know why not? We can come up with guesses. Certain is that certain mailers that are out there today behave differently in that situation than others do. Mainstream mailers. Regardless have what it may or may not say in the spec, we know there is a difference. It may not even be clear in the spec, because perhaps this case was not well spelled out. And that's where e-mail, one of the oldest services on the net, the one for which wildcards were basically designed. For a new service we don't know that this is going to behave in any way properly. And we do know that it's all but impossible, until much further development work is done, to give the proper, desired typo correction semantics, which I agree many users would like.

I don't quarrel with the call, I quarrel with the implementation. Conclusions are, the Internet is built on a set of architectural assumptions. This architecture has encouraged innovation by letting the edges do things freely without worrying about what the middle is doing. Breaking the architectural model will cause unforeseeable failures, both in less common existing software, and for future innovative ideas.

Question, Steve?

Steve Crocker: good, thank you.

People on the phone? People on the panel? You're going to get off easy, today, Steve. Questions from the floor?

Steven Bellovin: very easy.

Steve Crocker: is this a lunch time effect? Is everyone so compelled that there's nothing to chew on here?

Steven Bellovin: well, thank you.

Steve Crocker: thank you.

Mr. Klensin, you have the unique situation of actually starting early.

I'm sorry? My apologies. You sent this to me, right?

Draft slides for tomorrow. There we are, wildcard innovation. I think we're okay.

John Klensin: since I've stirred up a lot of curiosity, I'll ask the forgiveness of those people who don't know what I'm -- will not know when I get through saying what I'm going to say technically any better than they understand it now. The mail protocols require that there be-- that if a DNS MX record is found, that one look at the MX records, and if any of the MX records resolve, one stops.

That's clearly required in the current version of the e-mail transport standard, but it's currently required because there was -- but it's clearly required there, because there was a little bit of an ambiguity in the previous version of the standard for using MX records, which resulted in a number of then-widely distributed e-mail clients and sending servers, going all the way to the MX, down to the bottom of the MX list. if they didn't resolve, going off and pulling the a record and treating it as the lowest preference MX.

If one does that, and there are some-- there were some popular servers about five years ago doing it. And some of them are still very much in circulation. Then a tactic based on assuming one will run out of MX records will not work for that particular set of isn't clients -- SMTP clients. Are they in violation of the standard, yes. Are there a lot of them, yes. And that was the source of not work comment. Details on request but offline.

Mark Kosters: it would be good to quantify that again. You can say a lot, was it? What does that mean.

John Klensin: early versions of Microsoft Exchange.

Mark Kosters: okay.

John Klensin: and all of the versions of Microsoft Exchange before it became Outlook, and early versions of Outlook. In addition there are a bunch of other things but there's probably enough to quantify it.

I'm going to talk about, today, now that I've gotten over my quickie comment earlier, is the general theme of how innovation happens on the Internet. This discussion will overlap somewhat with what Steve had to say, but I'm trying to look at it from an applications writer. And from the standpoint of what happens at the DNS.

It's important to understand that part of what has made the Internet important to all of us is that new protocols and applications introduced fairly regularly. Some of them become big public hits. Many of them don't. Some of them are designed for, and used in fairly localized environments. Within an enterprise, within a cluster of enterprises for some particular purpose. Some of these new applications have the hope of becoming the next killer app and making everybody rich. Others don't have that aspiration.

Historically, because of the edge situation which Steve talked about, there's been no centralized impediment and no regulatory impediment to doing this. Unlike other network environments we've seen in the past, the Internet does not require anybody to come to a central authority and get permission to deploy a new protocol, because there is no central mechanism.

And one should for that reason avoid having a single organization as the gatekeeper for progress. for many years, we've been using DNS names and not addresses. It's better for users, addresses are hard to remember. It's better for understanding what's going on, which is why we have arrangements about who has rights to what -- we have rights to who has a right to what domain names. It's more stable. Change of ISPs, change of sites, changes, addresses, but names can adapt. As soon as one uses the DNS that way, it becomes part of the critical infrastructure.

I'm probably about to say the obvious to this audience, but new applications breed new companies and new developments.

If we centralize key functions, railroad approval or changes for new application, it becomes difficult to develop the protocols and applications. Most network traffic today is web, e-mail and maybe file sharing. But we can't look at something which says there's no or little usage on a particular port or protocol. And believe that that's relevant only if we do not care about tomorrow's application. Or about an application used in a business or enterprise in some isolated environment in the network. Or not so isolated.

The stability issue here is that if you're going to have new applications and develop them, it's vital that the infrastructure, including the DNS, behave in a stable fashion and in a predictable fashion. Maybe those are the same thing. Right now, we've got a prediction anomaly. Can an application depend on getting no domain, if there's no registration? Well, for about 200-some-odd domains, maybe 250-odd, the answer is yes. For some few others, the answer is no.

What does that mean if you are trying to construct a new application, dependent on this behavior? Well, you can assume there are going to be fictions out the there. If there are fixes and workarounds, they will keep things from working because they make it unpredictable in a different way.

As Paul mentioned this morning, if we get into a situation where the answer you get depends on where you ask from, we know long no longer have a DNS with universal references. If we get different behavior in different domains, even if the behavior is stable within those domains, we get a situation where the application has to worry about what the answers it gets means based upon what domain it's asking about. Makes it hard to write new code. Or keep old code working in an acceptable and predictable fashion.

One can adopt a workaround to that, have in every application a table of domains and knowledge about how they behave. The difficulty is that behavior may change. The bigger difficulty is that if such tables come into being, it makes deployment of new top-level domains very nearly impossible. Because what we know from years of experience, is those kinds of tables do not get kept up to date. These are, or at least seem like subtle technical issues.

But they're ones that have a direct impact on the user experience. Potentially on the economy. And on the general perception of the stability and usability of the network. Can you have a registry database, the registry-based database -- excuse me -- a directory based on a registry database without impacting the infrastructure? Yes. If you adopt a theme that there's a way of getting to that, that requires users who want it to ask for it explicitly.

A question which could be asked on behalf of the users by browsers or other interface software. Then certainly there's a way to do that. We've got lots of experience with naming conventions. We all know about www. A naming convention which says this is how you find a search engine for a particular directory, for a particular registry, Might at least technically be a very interesting application. And it doesn't challenge the infrastructure. In either event, ultimately users have to be trained. They have to be trained, make typing errors if they want to find something, or they can be trained to go looking for something specific if they want to find something. Probably the latter is a better approach. And it doesn't threaten the infrastructure.

We have another problem coming along, called internationalization. We're heading into a team when most of the users of the Internet are not native speakers of English. They seem to want their error messages in their natural language, their domain names in their natural languages, their browsers in their own languages. Probably that's reasonable.

The introduction of internationalized names in the DNS for reasons that are probably a little too complicated to go into here, creates a lot of opportunities for those populations. But also raises new risks. Risks arising from lookalike characters. Users are used to looking at URLs in browsers and assuming what's underneath the links is actually that link. The link they expect to find.

eBay has been having a terribly interesting problem lately with people sending out pieces of e-mail, which claim to be from them, look like they route the user to an eBay web page, send the user to somewhere else that asks a lot of questions that you don't like to give away, unless you like identity thieves. User has problems telling.

Can an e-mail program or browser be designed to work around that? Yes. Do I expect that will happen? Yes. But wildcards turn out to be a challenge to doing that. And turn out to be a worse challenge to doing that in the Internationalization case. Because it turns out that most of the work on internationalization argues that you should have some names which should not be permitted to be registered if other names are registered. And if every name appears to be registered, then that system doesn't work.

The Asians who spent a lot of time working out a system like that are very unhappy right now. I was telling a friend of mine about some of this the other day, and going over a piece of this paper. And she had an interesting comment. The comment was, with all of these problems it's amazing the Internet works at all. And I thought about that. And she's right.

But this particular piece of amazingness comes from the fact that the network in many respects has been designed to be very robust against many kinds of abuse. Abuse of the protocols, abuse by badly-written programs, abuse by particular users, as well as simply the routing difficulties of a link going down. We've discovered the thing is robust enough that misconfigured e-mail systems work reasonably well most of the time and produce good diagnostics when they don't. Even though the configurations should not occur.

We've seen configuration programs which select defaults based upon names and that usually work even when the names aren't right. It may not be a good idea, but we dare not break them because there are lots of them out there. And doing something which requires thousands of people to spend an hour or two apiece installing patches gets expensive across the economy. Hundreds of thousands or worse.

E-mail and fax and voice over the Internet are actually fairly secure and private as long as you can trust the sending system, intermediate ISP and the receiving system. If you can't trust the DNS to yield the right names and links, that breaks down. Much of our protocol design has been based on something called -- has been based on it uses a meta rule about all Internet protocols.

We call it the robustness principle. And it says the things that you're sending things out need to be very conservative about what they send, regardless of what the letter of the protocol says, or regardless of how one can carefully read the protocol to define something without explicitly permitting it. And the things that you're receiving, things need to be conservative about what they accept. We've ended up with a very stable infrastructure which includes predictable responses to unexpected conditions.

This principle of conservatism is part of every Internet standard protocol, even if it's not explicitly stated in every paragraph or in every document. And that a behavior is defined does not make its use reasonable, appropriate, or even permitted. If we adopt a rule that something in a definition of a protocol makes it reasonable, appropriate, or permitted, we will squeeze all of the robustness out of the system and the network will probably stop working the next time somebody does something strange.

Ultimately, the discussion we're having here is not about preventing innovation, but about continuing to enable it for new applications by providing a stable and predictable base on which those applications can be built, a base which includes the DNS and predictable behavior of all domains.

Are you going to let me off as easily as Steve got off?

Steve Crocker: no. Any questions from the phones? There's one that came in that relates to your first part.

You talked about the old mail applications. What's your estimate on what the -- the time it takes for those to die off and the replacement cycle?

John Klensin: if you want to talk about half lives, we're probably over them already. But the interesting property of half lives is they have very long tails. We know there are some large corporations which got stuck on NT4 and who don't like newer things and who aren't moving.

We know there are some large organizations which got stuck at early versions of Exchange Server and have not been willing to deal with the transition. Similarly, some early versions of Lotus Notes had this problem. It's very, very hard to know. We know that in a lot of environments, that kind of software rolls over when computers roll over. Computers roll over very rapidly in some environments. But we are still shipping things to third -- to developing countries which don't run new operating systems because they're not big enough. And this has become a worthwhile method for developed countries to spread Internet access to less-developed areas.

So in some sense, we're still exporting some of these bugs.

Steve Crocker: let me take you on on this for a second. Just I'll play devil's advocate here.

Can't you argue that if we can't make innovation because we have to worry about the legacy software which is out of spec, and that spec -- and it stays there for a very long time, where is the point at which you say, "well, there is a balance between adhering to these things which are obsolete and out of spec and being able to move on at the cost of the disturbance that that costs"?

John Klensin: it's an argument I make regularly. And the answer is, there are always trade-offs.

If this were a situation in which this behavior had always been clearly invalid, one would clearly have more sympathy for that argument than in a situation in which that behavior was questionable until three years ago, and then became clearly invalid.

It's especially tricky with e-mail because of the number of applications which get built using e-mail as a transport. And those are things in which the mail transport environment is not directly visible users and may be sitting in embedded environments that may be really hard to get to and change without unwiring major boxes.

But the answer is, it's always a trade-off. I don't know how you work the answer to that question.

Steve Crocker: that's an awkward position for us to be in, don't you think, that here's an aggressive, young struggling company that Verisign is trying to, you know, make forward progress in a chaotic, troubled world, and here we are saying, "well, you have to support Microsoft's NT4."

John Klensin: I certainly wouldn't want to support -- well -- (laughter.)

And I think Microsoft would stand here and say they prefer not to support NT4, either.

Steve Crocker: they stopped supporting it before they shipped it, if I'm not -- I'm sorry.

That's a separate question.

(laughter.)

John Klensin: yes.

The difficulty here is the -- from my point of view, the issue with the ports we don't know and the protocols we don't know about and the one which comes along next is a much more significant issue than this particular can we get away with this on e-mail given that we fixed the MX record situation, which is why that comment was not part of this talk in the first place.

I think that, ultimately, if we're smart enough and if we spend enough time at it, we can figure out ways to squeeze out almost all of the unreasonable error cases from interactive use of HTTP and interactive use of mail, because we understand them pretty well. There will be marginal uses of HTTP and marginal uses of mail, which we probably won't get. And there we have to make the tradeoff as to whether those are important enough.

But the protocol we can't predict. the next killer application. Which if it queues things up and gets something which isn't responding or which resets will queue them up forever rather than telling the user that there is no DNS record, that's a much more serious problem. Because that, and not the situation with MX records, which is a legacy software problem, is the impediment to innovation.

And, frankly, I'm much more worried personally about the impediments to innovation going forward than I am worried about the legacy software. The comment about the MX records is that if you assume that putting in an MX record that way will solve all of the legacy software problem, that's not correct. And I don't know where you draw the line there.

Steve Crocker: thank you. Ken.

Ken Silva: yeah. I've got a couple of questions.

Why has this issue not come up -- if these problems are so fundamental or potentially prevalent, why is it that it hasn't come up yet in all of the other top-level domains that have used wildcards? And why is this only now becoming an issue if these problems were really as severe, I think, as the last two speakers have suggested, I propose -- or I guess I ask you, why have they not materialized sooner?

John Klensin: let me pick on two domains as examples.

One of those examples is museum. They've got a bunch of registrations at the second level. They've got some fairly strong norms about how the second level and the third level are organized. The nature of their users and user communities is that the odds that new protocols are going to be invented there rather than at other places in the future are pretty low. Certainly they have not been so far.

New domain, low-population group of registrants who are focused on using web pages to deliver a web page-like service from their second- and third-level domains to their users. So the problem hasn't shown up because we haven't seen the innovation there.

We've got some other domains -- let me pick on nue. Nue has been marketed in interesting and creative ways, but has been marketed again in interesting and creative ways to folks who are trying to establish an identity for web sites. They don't have the contractual situation that museum does and the prior notification of every single registrant that that wildcard is there. But they have tended to not draw Internet innovators but web site innovators and people looking for that particular name, for whatever reason.

One of the problems you've got with com is you have a bunch of registrants who are innovating, a bunch of registrants who are large, corporate enterprises who are doing interesting things within their networks which may not be isolated in a single island. You've got a bunch of registrants -- I don't know what percentage -- who are not running web sites at all but are using their registrations for other purposes. And for that reason, the scale problem is much different, not just the scale problem in size, but the scale problem in kinds of uses.

If we were looking at this situation five or six years down the line and, for example, nue had succeeded beyond its registrar and registry's wildest expectation and had even as few as hundreds of thousands of very diverse registrations in it doing innovative things, we have a problem there and it would be no different. But you're exposed because com has been the catchall since the time we set up the DNS. And as a consequence, we've got these large-enterprise domains in com and almost nowhere else except a few ccTLDs that have not adopted wild cards.

And we've got these innovative applications springing up in com because of that same history. Same comment with net, only at a slightly smaller scale. Same comment with org, only on a slightly smaller scale, I expect the same with edu, only on a slightly smaller scale.

Ken Silva: yes.

But isn't it our responsibility to make sure that the DNS system -- as I heard suggested earlier, the DNS system must consistently respond in the same way. In fact, that's not the case today. Some TLDs respond a little differently than other TLDs.

John Klensin: if you're asking me if it's unreasonable for you to be sitting here as the focus of this attention and not those other TLDs which went off on instantiated wild cards, my answer is, yes, it is unreasonable for you to be sitting here taking all the heat.

Just my personal opinion.

Steve Crocker: not to disagree with you, but there are two different forms of consistency, just for everybody's.... One is, for a given query, whether you get the same answer, no matter which ISP you're sitting in, which is how this came up earlier today.

Steven Bellovin: that was certainly my primary meaning.

Steve Crocker: and then the variation of that question in this context is, when you ask a similar question but in different TLDs, whether you get the same handling of that. And those are both reasonable questions, but they are somewhat different dimensions of instability, if you will.

John Klensin: one of my worst nightmares as an application software developer is having to carry around tables in my application which list all of the TLDs and how they behave so that I can adapt appropriately to each different kind of behavior for each different kind of TLDs.

And that's a nightmare whether it's about international domain name coding or character sets in use or wildcards or any of a whole series of other things that, in principle, different TLDs could apply narrow readings of the standard and say, "we're going to do this in some other way."

Steve Crocker: but couldn't you just look up those differences?

That's a joke. That's a joke. You're supposed to understand that's a joke.

(laughter.)

Ken Silva: I mean, it does seem like we might be sort of painting ourselves into a situation where -- I mean, I look back in -- you know, five years ago. And I think that if it were left up to a large part of industry, we wouldn't -- computers would never have entered the 21st century, because it was -- it was too hard to get them Y2K compliant. I don't think this is anywhere near that issue. And I think that the -- most of the top-level domains, wildcarded or not, do respond in a consistent way to applications that are responding to standards.

John Klensin: if I can be guaranteed that my users -- as an applications writer, if I can be guaranteed that my users will never make an error, then you are probably correct. Unfortunately, I don't know how to get people to do that.

Steve Crocker: yeah, if I might, let me just pick up that thread, Ken.

I think the -- and, please, push back if I'm mis-characterizing it.

But I think what's emerging out of this is that there is a standard error code, this RCODE 3, which for the domain name doesn't exist, which is now being transformed into a sub- -- the appearance of a legitimate address, and then a query that goes to that address, never mind the non-registered aspect for spam.

Let's set that aside for a second.

But the protocols that are attempting to accomplish something and will make a connection to that address, and then for every protocol, current or future, there has to be a well-defined equivalent response that says, "well, you reached this place, but this place does not actually exist." if that were a standard part of the architecture and that the architectural guidelines that every time you create a new protocol, there has to be a section that says, "fit within the protocol the notion of you got here, but this doesn't exist."

We have standard parts of architecture in other aspects of our design ethic. We -- some years ago, we started writing a security consideration section of every RFC that defined a protocol, for example. So there's a series of those kinds of sort of meta issues that are layered across the entire architectural and design landscape.

But I don't think that we have yet a -- an error code within protocol that says, in essence, "you're here, but there is no here here, so this is your error code that says, you know, you arrived at the wrong place." if there were, then you would have had a very easy time -- not you, but Scott and Matt, I guess, or whomever -- would have had a very easy time getting it exactly right the first time of how to put those negative responses in at the location.

So that seems to me to frame the question of these error codes and the architectural issues that go with it. At least that's where I'm coming from.

John Klensin: yeah. We do not have a consistent architecture by which every application which happens to be listening on a port can say, "no, forget about it, I don't do that." We don't have a standard architecture for ports for which nothing is listening coming back with a standard and predictable, “no, I don't do that response.“

If we did, then it would be possible to build something like the SiteFinder server to respond to protocols it had never heard of before. But we don't have that architecture.

Ken Silva: right. Okay. So right now, it does simply reject the connection for anything other than HTTP or SMTP, okay. It's rejected in a fairly well-documented way.

I guess what I was kind of hoping to hear from the presentations was not just so much the theoretical what could go wrong kinds of things. The service ran for a number of weeks, a couple of weeks, and, quite frankly, we didn't experience the -- nor did our users experience, a lot of the catastrophes we're sort of hearing are theoretically possible today. And we've talked to a number of people, and we're just not seeing all of the odd instabilities that are claimed, quite frankly.

John Klensin: I think that's predictable. And let me try to explain two reasons why it's predictable.

The first is that if we're worried about a protocol which first gets developed and deployed on the Internet next month, you're clear we've not seen it the last month. And that's part of this innovation problem.

The second is, the vast majority of the protocols which are doing special things and which you've not heard of, or have not paid attention to because it's down in your 5% or 10% or whatever that number is, are being run in relatively well-behaved environments, being run out of scripts, run out of other environments like that in which nobody gets domain names wrong.

And if they don't get the domain names wrong, then you don't see it. But in an attempt to expand the use of those protocols or write new scripts or do new developments, suddenly causes things to crash and burn. And as someone pointed out earlier, causes things to crash and burn in fairly obscure and hard-to-track ways.

The best examples would be things which behave like e-mail but aren't e-mail. You're not at the moment -- easy problem to fix, but it makes a handy example. You're not at the moment tracking the submit protocol, which is an SMTP clone on another port. If someone comes in with submit protocol and gets a reset, the server for that protocol, the sending client, sending SMTP on that particular protocol and port is going to queue. And it's going to queue and retry. It's going to queue and retry for a minimum of 3 days and a maximum of -- it depends on how paranoid you are, but a week and a half is not uncommon. That's a very, very seriously delayed error message problem.

Now, again, why aren't you seeing it?

Well, you're not seeing it very often, because in the vast majority of cases where somebody is configuring an e-mail sending program first top, like submit, they're configuring a name into an e-mail client and then leaving it there forever. And those don't roll over very quickly. And as a consequence, those errors are not made very quickly. If you've got an e-mail client which runs through several servers if it can't make a connection on the first one, then the odds of that problem goes up. There are some e-mail clients like that out there but not a lot.

The protocol which comes along which you cannot predict, as I say, the submit problem is easy to fix if you have an SMTP (inaudible) control. But if you have a protocol which has that kind of queuing behavior but which you don't know about, and there are a number of overnight batch file transfer protocols which work exactly that way, then there's a problem.

And do you see it?

It depends on whether someone is rewriting the script and makes a typographical error or whether the scripts from last month continue to run smoothly. That's the difficulty with this, which is because you're trapping an error behavior, to prove that it's working correctly is a universal negative problem, and none of us are good at solving those.

Ram Mohan: this is Ram.

I also wanted to add that just because you didn't hear it doesn't mean it didn't exist. I think there has been some data that has been presented, albeit it's very little so far. But there has been data presented that shows that some things did break. So I'd be reluctant to characterize that as odd or as corner cases. Clearly, more data has to be collected. And all the parties that are involved, including Verisign, you know, should go publish that data. But I am reluctant to say that -- or to agree that that was -- that what has been presented so far is purely theoretical or only corner cases.

Steven Bellovin: I'll give you a firsthand example.

For reasons I did not and do not understand, I could not get to the SiteFinder site from my office. It was not local blocking. I could watch the trace route going far out looked like past the boundaries of our ISP, and then it didn't work. I could get there from the same time from other sites on the Internet. I could not get there from my office. I do not know why. I can show you the trace routes.

Ken Silva: so what broke because of that?

Steven Bellovin: well, if I had a mistyped domain name on the e-mail, I would not have gotten an answer until it timed out several days.

Because the mail server would have said, "oh, TCP is not connecting. Time out trying to connect. That's a clear temporary outage. Let me requeue and try out later." I tried several days and I could never get through from my office. Again, I do not know why. We are a well-connected site. It was not us; it was not our ISP.

Steve Crocker: your well-connected site is AT&T?

Steven Bellovin: yes.

(laughter.)

Steve Crocker: any other questions from the panel members here?

Anybody on the phone want to chime in at this point? And from the floor? Ah.

Richard Smith: sometimes I still develop software. It's been a while, but I still do a bit of development. And --

Steve Crocker: you should identify yourself.

Richard Smith: yeah, this is Richard Smith.

Steve Crocker: thanks.

Richard Smith: and while you're doing your slides here, one of the things I'm working on is a tool for tracing sort of hacker web sites and this sort of thing. It's all classic stuff, trace route and ping and stuff like that. So what I want to be able to do in that tool is detect the difference between a nonexistent domain versus the server's not there. I need to make that distinguishing thing. So it seems like that's going to be a lot of what application developers are going to want to do.

We're going to come up with these sort of ad hoc rules for detecting the distinguishing -- distinguishing the situation. That sort of way around this innovation issue that you were talking about, do you expect that to happen?

Do you see us -- all of the programmers going out and coming up with ways of still being -- detecting nonexistent domains, to provide users with good user messages?

John Klensin: I think Paul Vixie's talk is evidence that's already in progress. The difficulty is that every one of those fixes potentially causes another problem. And the difficulty with the approach you're suggesting, which I am certain you can make work, because you seem like a smart guy, is that it's ultimately based on a set of heuristics. And things based on sets of heuristics will give you good indications as to what you're trying to find, but won't give you answers.

Richard Smith: okay.

John Klensin: something will sooner or later come along and force you to keep retuning those heuristics. That's not good uses of anybody's time unless your business model is to force the users who upgrade frequently, in which case you may love it.

Richard Smith: that brings up a follow-up for Verisign. Would they bless the method of detecting a nonexistent domain name and a server not being present?

It seems like a pretty reasonable request as an application developer to be able to get good error messages to people. Right now my hacker tracker tool, if I mistyped the domain name, it thinks you guys are hosting the hacker web site. I'd rather not make that mistake.

Mark Kosters: there are ways of actually detecting whether or not a wildcard is present now; right?

And it's a fairly sort of simple set of DNS queries one can do to actually see whether or not the synthesized response -- actually, just query directly for a wildcard and see if subsequent queries actually match that particular a record. Then you know you hit the wildcard. So that's one way of doing it.

Another way is actually something that's been brought up in the IETF before, certainly it would be nice to have a protocol enhancement. Some people feel this is a good idea to actually put into the protocol, saying this is a synthesized response. There is none right now.

And Paul can correct me if I'm wrong. So if -- so there's various avenues that one could take going forward. The nonregistered address. We don't have that architecture. Couple of weeks. We didn't experience the, nor did our users, experience a lot of the catastrophes, we're sort of hearing are theoretically possible today.

We've talked to a number of people. We're not seeing all of the odd instabilities that let me try to explain two reasons why it's predictable. If we're worried with a protocol that first is getting cleared out deployed.

The second is, the vast majority of the protocols which are doing special things, you have not heard of, or have not paid attention to, it's down in your 5% or 10% or whatever that number is, are being run in relatively well-behaved environments. I'm reluctant to pay or agree theoretical, or only corner cases.

For reasons I did not and do not understand, I could not get to the SiteFinder site from my office. It was not local blocking. I can watch the trace route going, looked like past the boundaridries of our ISP and it didn't work.

Can I get there from other sites on the Internet, I don't know why not. I can show you the trace routes. A clear temporary outage. I tried several days and I couldn't get through.

Richard Smith: another question is Microsoft could start doing this in Internet Explorer, then. Detecting the synthesized DNS.

Mark Kosters: they certainly could, absolutely.

Richard Smith: thank you.

Steve Crocker: if I understand the solution, it is doubling the queries, you have to precede every query with the wildcard query, or cache it.

Mark Kosters: caching would be a reasonable sort of thing to do.

Steve Crocker: I went down this path in my mind, while we were thinking about this. What is it that controls, you guarantee in the wildcard, just at the edge of what I understand, do you guarantee that the wildcard response remains stable during some period of time? Whatever its time-to-live is, it won't change out from under you?

You can imagine you get a different IP address for a range, that you would respond to, that would not be an unreasonable implementation, that makes it a great deal harder to know which response is, in fact, synthesized and which one looks like an record IP address.

Mark Kosters: well, okay. We're talking kind of theoretical in terms of what application developers actually could do? They may come up with smarter solutions.

For example if you do a query for a wildcard and you have multiple a records coming back, they'll list up and you can choose off that list. Or match off that list, for example. But this is something that is certainly up for discussion in terms of what is the best way of dealing with this. I think many of us are interested in hearing what the best solution is.

John Klensin: two observations. Two depends on there not being any legitimate labels with the same IP address as the wildcard floating around, doesn't it? I can't tell if SiteFinder dot something dot com is distinct, is a real record or wildcard. By that algorithm.

Mark Kosters: that's correct.

John Klensin: second closely related observation, if we employ DNSsec we can use that to tell the difference between the data record and the real one.

Not so fast.

Mark Kosters: actually no.

John Klensin: thank you, I thought we could.

Mark Kosters: I think there's definitely some things that one can on a protocol level, to mark this to do that. One would have to go through the IETF to do that.

John Klensin: the difficulty, even if it's done, it means we have to trickle information up. We have to trickle applications up to make decisions rather than trickling it only to the DNS interfaces to the applications. That gets back to another one of the kinds of trade-offs Steve was talking about.

How much burden and expense is reasonable to impose on every application writer and every deployed application in the Internet, in order to put this thing to work properly in the associated workaround.

The question is whether it's reasonable Internet stability to deploy something that requires everybody else's deployed applications to start installing workarounds.

Alan Davidson: Alan Davidson.

I wanted to ask a broader question. The question is for you, John, and Steve.

The question is, so what, exactly, do you suggest that we do about all this? And, I mean that from the point of view of saying the values, the issues that you raised about the architectural issues seem extremely compelling, extremely important.

From a user point of view. A lot of the architectural features in terms of promoting innovation promote a lot of things that users care about like freedom, and other kinds of things like that.

And I guess I'm curious about how optimistic you are about our ability to define clearly that which is permitted in registry services, or even more generally. And what process we're supposed to use.

This in many ways seems like a process failure, process question. I'm not sure what the answer is.

Steven Bellovin: I could cop out and say that's not what this committee is charged with. But seriously, I'll turn that cop out in a serious answer. It is an interesting policy question. And of who should have the right to decide what things go where. I'm quite convinced, personally speaking, that it needs to be an open process. And not for the-- let's look at it very concrete terms.

The biggest typo correcter response is Microsoft's, it's Internet Explorer. Verisign has their own. With the standards existing the way they are today, they're basically incompatible. You have one or the other in some sense.

Yes, we can figure out ways to get both, but basically one disabled the other. And I don't know a good way, value-freeway to decide which one, hence which company, should win. Other than it's got to be some sort of open community process.

Alan Davidson: I don't think you can duck the question. It may not be clearly a question that's for you to answer. But it is a question, the answer closely intersects to some substantial degree what you talk about.

Steve Crocker: so I get to say the famous words, it depends what "you" means.

Here, today, we're the security and stability advisory committee, our focus is on that matter. The question you're drawing falls in an interesting area. Because if we view our charter very broadly, you can argue that there are stability issues, and that this falls within it.

But it's fundamentally a process issue, that you're raising. With the process issue, is the questions of openness, like Steve Bellovin said, questions of who has what prerogatives, is this a private service that Verisign is running. And that everything else is sort of interference.

Or is this basically a public service that is owned for the public good. And that that's where the control should lie on that. And there may be a spectrum there of some sort.

This committee, and this setting, is clearly not large enough to encompass all of that. But I'll speak for myself, but I think the rest of us as well, strongly agree that that's an important question. And is not a question to be escaped and ducked. It's just not one that can be dealt with solely within this setting. Nor we ever intended to. We are part of a larger process that exists within the overall community.

Alan Davidson: I appreciate that. I was taking the opportunity, because we had two presentations that squarely brought it up.

Part is depending on where you all think the solution space may be, might make a difference in terms of how -- what process is used.

Anyway.

John Klensin: let me suggest -- Alan may not want to leave the microphone yet.

I suggest this is a stability issue. To the extent to which part of what we're trying to preserve on the Internet is the ability to innovate and deploy new applications, the notion of a very stable pattern and a DNS which behaves the way people have been expecting it to behave in terms of error code returning and stable and predictable behavior or the last many years is pretty important. If we don't have that, the applications become vastly more difficult to write. And that discourages innovation and is a stability issue.

Unless we decide that kind of innovation, the absence is not important. If I may take an example, to move down a level, and doesn't address the DNS, I worked for an ISP some years ago whom some people thought was in danger of becoming large and powerful enough in its traffic-carrying capacities to exert monopoly power on the network. And that ISP could have stood up and said, you know, it would be much more convenient for our business to change the format of an IP packet a little bit.

And everybody else had better adapt. And if they don't adapt we're going to disconnect them at the peering points and we're going to tell all of their customers how much better off they would be connecting to us directly, because we are the big guy and we have all this connectivity.

That's a situation, if anyone had been able to control in the interests of stability, and predictable behavior of the network, no one would have tolerated. Or the other ISPs would have figured out a way to present the requested packet format at gateways, but nothing else. And that ISP probably would have lost out in the marketplace, or not. But it's not behavior that the extent to which we want a single international seamless connected Internet. We want to be in a position of encouraging. And that gets us both the policy problem and the technical stability problem, again, of how much conversion effort and workaround effort and adaptation effort somebody gets to impose on the rest of the network by unilateral change in how the protocols behave.

Steve Crocker: so I agree with you.

Let me come back to the answer I was trying to give Alan before. We're in an area where some people will say this is the stability issue. And other people will say, no, this is a business issue, or this is a governance issue or this is some other thing, it's not a stability issue per se. That raises some very delicate issues about, okay, do we in our activities here, err on the side of saying, if it is possibly a stability issue, we have to take that up, and it's ours? Or if it's only certainly a stability issue, we have to take that up. And the way it happens in the things where some people will take one position on it, others take another posture on it.

Having made the problem hard, let me now try to make it simple. And the answer is in a way simply yes. I think that it is incumbent on us to address the problem and not own it. To contribute to it, with the understanding that it is both a well-framed question, but also a question of whether or not that belongs in this set this way or set that way. Then we have to, and we will, engage the rest of the community packaging up our perspective on if it's our problem here us how we think about it. But we don't necessarily know that it is only our problem.

I recognize that's a compound, complicated construct, not the kind of thing that one likes to float inside the beltway. But nonetheless, sort of the complexity of what we deal with.

John Klensin: here is where I get to be grateful I'm not sitting at that table.

Any other comments?

Harold Feld: : Harold Feld.

One of the aspects about the end-to-end implications for this, and which goes to the question of stability, is a question of has Verisign, and I think people brought it up in different context, does Verisign prevent other people from doing the same. And that question is being asked in a couple of different ways.

One was Steve's comment of, well you have a tug-of-war between IE and SiteFinder. And to some degree SiteFinder preempts IE. But Microsoft, being smart, can give you a patch which will let IE trump Verisign. That's one way of having the question of is it blocked or is it not blocked.

Another is, if Verisign has a capacity to say we are going to deploy this in a way that no patch will let you get around what we're doing, without causing you such a big headache, that you'll never want to do it.

So, you know, I hate to keep, you know, pumping up the hard definitional questions of what is stability, what is the impact on the architecture. But it just seems to me one of the things this committee is, unfortunately, going to have to figure out is if you are looking at this in terms of does this break end-to-end, is the dot com and dot net as they're constituted such a core infrastructure that we can't allow change or we would not allow change at all, TLDs preferably, but dot com and dot net draw fire because they're such a dominant part of the market.

But how are you going to draw, you know, the line between on the one hand you can look at this and say this created a lot of innovation. We've seen lots of people trying new things in the last month, some have been inspired to try more. But is that disruptive?

John Klensin: did you.

Steve Crocker: did you say this was an economic stimulator causing chaos and getting people to --

Harold Feld: : I said that's one way to look at it. I didn't say that's what I said. It can be destabilizing if we get into a tug-of-war between Verisign and Microsoft. I don't particularly want to live in the middle of that universe.

Steve Crocker: why limit it to Verisign and Microsoft? Once the opportunities are there to get in the middle and sort of reinterpret what's going on, I think Cisco has a good opportunity. They have the software that handle these packets going upstream and downstream. Not all of it, but most of it. Why can't they reinterpret and add value to it?

Harold Feld: I don't disagree. I think all I'm saying is in terms of what you're doing, let me put it this way, in terms of looking at stability, those are part of the questions you're going to have to ask. You can't limit it as some suggested to what got broken over the last couple of weeks. And is that tolerable to end users, and to people in the Internet. It's got to be a broader question of what are the broader rules of the game under which we're going to proceed.

Steve Crocker: I have a question that's come in from the field here. Let me just read it. And I'll address it to you, John.

It's unlikely that the SECSAC would be concerned about my using synthesis wildcard synthesis for the domain, I'm the registrar of that domain, I can only hurt myself. Where the concern is when a registry acts as an independent third process, at the DNS root, the reason is multiplication of the potential damage. This makes it different from a random level service.

Would John or anyone else on the panel care to comment on this perspective. The answer is "not me."

John Klensin: I would draw one small distinction that the person asking the question did not. Which is the-- one of our interesting distinctions to the DNS, which keeps getting lost.

Is the difference between a domain, which is under the control of the enterprise which is using it, or the individual, and a domain which is being operated as a service for customers, and that domain is -- has a customer-vendor relationship with the people, with the people deploying the host in that domain. Rather than a part of the same enterprise relationship. That distinction is inherent in administrative domain design of DNS, and is why we've always been a little sensitive about domains closer to the root.

A different way of stating that rule is that it does make a difference whether I shoot myself in the foot or somebody else does it for me.

Steve Crocker: other people with questions or comments on this subject?

Good. Well, we've come to the place where we carefully left things open and undefined, to see what emerged out of here. We had some interactions earlier where we had to be -- we had to curtail the discussions in the interest of time.

I'm prepared to choose topics, or pick up old topics, and structure the time. Or if we've come sort of to the end of the useful time, then I'll make a few closing remarks. And people can be out of here a bit early. This really has to be driven by those of us who are here.

Anybody on the floor want to open up a topic, or reopen old topics?

Getting action here.

Huey Callison: somebody asked me coming back from lunch, what is the big deal. To Joe End User, this type of domain gets a different web page. Who cares?

Obviously the people in this room care or they wouldn't be here. I submit to you, to Joe End User, it's not a big deal. And here is an idea.

Because the scope of the people for whom this is going to have a negative impact, applications are going to break, is comparatively small, I don't like web pop-ups. I have this cool HTTP proxy that just sits in front of my web browser and every time it gets an ad, the ad goes away. When I was running a mail server, I subscribed to a third-party DNS-based blocking list. Run by Mr. Vixie at the time. Every time a connection came in it would look at the list and the connection would go away. In both of those cases, the lists are maintained by a third party.

You said in your presentation that keeping a table in your application is going to be problematic. It's not problematic if it exists within an application and is updated by a third party. If Verisign says this is what we're going to do, if you want to work around it, here is how you work around it. They could produce the application. And then the people to whom it matters, could address the problem.

Is that an unacceptable burden on the rest of the unit?

Steve Crocker: do we want to sometime you a late discussion on that?

Go ahead, Mike.

Mike St. John: one of the fastest growing sectors of network information technology is the wireless space. One of the comments made in this whole process has been that we've changed a simple short error message in a 17k download. That becomes a direct expense to the end user, but that doesn't get filtered out by the infrastructure in the middle. Before the infrastructure in the middle could translate that error message into two, three characters.

I don't remember what it does with it. With the longer stuff, it's hard to tell it's an error message. At least one example, I'm sure we'll find more as we go along in this space.

Steve Crocker: go ahead.

John Klensin: John Klensin.

There are a series of questions like that. That I'd like to encourage you to think about in a slightly different way.

Let's pretend it's 1991. And we're asking exactly the same question about who gets hurt. And there's this protocol for which a port has been allocated by IANA for use, port 80. But nobody's ever heard of it. The server sitting at the end of this wildcard resets on port 80. Everybody who is trying to use that particular odd protocol discovers they have no idea whether the thing at the other end isn't there or they have managed to hit something badly, or the network is congested, or any number off they are things. Which that particular response can look like to a client implementation for that particular protocol. And if you follow that story so far, the answer is, we might not have a web today.

Ram Mohan: there seems to be a fallacy that the Internet is the web. The Internet is much more than just the web.

Chuck Gomes: : Chuck Gomes, Verisign.

I have a question, and I don't know if I drew the right conclusion from some of the things I heard this afternoon. I would at least like to throw the conclusion out. And see if I did or not.

And what I think I could conclude from what I heard this afternoon, is that it's okay, in fact we should encourage innovation at the edges, even if what the applications that are built there are not standards compliant. But at the center, we should discourage innovation, even if they are standards compliant. I don't know if that's a correct conclusion or not. But I think that's what I heard.

Steven Bellovin: certainly did not hear the part about encouraging standards violating innovation at the edges from me. The important difference, though, is if the violation takes place at the edge or if it simply doesn't do what I want it do do, I don't have to use it. I could avoid it. If I don't like what Internet Explorer does when I mistype a URL, I can run another browser other than Internet Explorer, which, in fact, I do, I run Galleon, a minority operating system platform at that, because I have that choice, because I'm at the edge and I can do what I want to at the edge and not have the decision made for me by someone else. That, I think, is the hit. If I regarded Microsoft's Internet Explorer as standards-violating, I wouldn't -- I don't have to use it. I don't really have a choice with TLDs and dot-com.

Chuck Gomes: and I would agree with you that that choice at the other level -- and I don't believe that SiteFinder limited that choice, at least for the providers like Microsoft and others, as has already been pointed out, that choice was still there. And that's good. That's part of the market working.

Steve Crocker: well, now we move into a fact issue. And the fact is that there was a change made at a place where only the registry operator can make that change, where there's no choice for anybody to propose, make any alternative changes or whatever. So that was a unilateral action on the part of the registry operator.

And for better or for worse, the architecture that we have at the moment is -- and that we've had with domain name system is that registry operator are in a unique position quite separate from ISPs, for example, one can go to an alternative ISP. But there's no way to go to an alternative registry operator when making a query, not when choosing who to register with, but when making a query. That's just a fact about that. And that is completely separate from any service that's offered around the edge.

I think this is just a very specific, clear-cut technical distinction that is -- it's not all of the issues involved but it is one of the core issues, and it stands out separate and apart from a lot of the other software issues around there. That's the one that leads to other broader questions about public trust and about contractual approval. But the technical issue is that the registry operator controls what comes back from a registry query.

Nobody else can offer an alternative to that except by then engaging in these very complicated and, you know, unseemly kinds of destabilizing things of intercepting it and undoing it. Then you get into a warring kind of state. But that seems to me just a critical factual point that is not a question of policy. I mean, there may be policies whether you should or shouldn't. But the fact is that's the point of leverage and it's unique and it's not open to alternative acts.

I'm sorry to get into a speech on that. But that's a clear point.

Let me take, at this juncture, the opportunity to read two statements that have come in from people. They go in different directions. And there are a couple of paragraphs. So bear with me.

Daniel Karrenberg, who's at RIPE and I think a founder of RIPE, one of the principals there, longtime, key person, has some remarks on data spillage.

And he says it's important to understand that the wildcard a records will affect all traffic directed at misspelled domain names domain names. The traffic will be sent to the IP address the wildcard points to. Depending on the protocol, the amount of data spilled will vary, but there will always be at least minimal spillage. This cannot be fixed. One has to trust the operator using the traffic not to use the spilled data.

I'll insert a little comment.

There may be other circumstances in which there's data spillage and there's trust issues. So this may not be unique in the -- across the network. But nonetheless, that's the point he's making, is that there is data presented at this site that would not otherwise -- or as the IAB put it, quote, an interception service with this kind of scope raises significant privacy concerns since traffic received by the interception service is pretty much by definition not going where its sender originally intended.

Stability, in general -- so that's data spillage.

Then he talks about stability in general, which he says is the most important point. Stopping the return of the RCODE 3 to queries for names not in the com and net zones is changing widely expected and accepted DNS behavior. This previous behavior is codified in a large number of protocol stacks and applications, most of them will change their behavior in turn, changing widely expected behavior will cause fixes to be deployed to reinstate the previously expected variety. A wide variety of such fixes will be deployed without coordination and will aim at particular symptoms. It is evident this is bad for stability. Therefore making the changes without so much as prior notice was irresponsible.

So that's Karrenberg. I have to say that's not my words.

Even when announced long before hard, this change does encourage local fixes which may very well lead to balkanization of the DNS, that is, users in different places on the Internet will increasingly get different answers to the same question. It's evident this is even further from the expected behavior and it's evident this is bad for stability. So in many ways, this echoes the points that have been made earlier in the day pretty straightforwardly.

We can have a discussion on this. I suspect Daniel is still listening. I don't know if we can engage him directly. But I felt obliged to read that.

The other message comes from -- let's see if I can pull this up here -- from Rick Wesson. Rick is actually on our committee comes from the registrar community.

He says today we've heard a lot of issues about the deployment of SiteFinder.

On October 28th, (reading quickly.) While SiteFinder only affected nondelegated domains, WLS will affect all delegated or registered domains in the Verisign com and net registry. I am concerned we will be in this room again on November 7th to discuss the implementation -- sorry, the implications of WLS.

He says I, Rick Wesson propose Verisign registry postpone the development of WLS until the community has had ample notice of the affects. A significant and negative reaction to SiteFinder, it is my opinion WLS will be as disruptive and the community deserves notice and the opportunity of debate on services that impact a large number of domain name holders. I'll note that WLS is getting a lot of attention in other forums besides ours. So I don't know how -- I'm just plain-old tired. So I'm not sure how eager I am to take on more things. In fact, maybe this is a point at which I'll insert a small note about we didn't go looking for this problem. We don't sit around waiting for the opportunity to get involved in some flap. This sort of came up and overtook us.

And I think it's important for everyone to understand that this came and hit us over the head and dragged us in as opposed to us, you know, licking or chops and saying, "oh, goody, we now have something to chew on." The community reaction, which there may well have been many people who were happy, there may have been, you know -- I don't know what Verisign heard. But certainly we were -- it was sort of imposed on us with the amount of messages and complaints coming in to -- and made it obligatory to pay attention.

So that's just sort of my statement on that.

Mark Kosters: I'd just like to point out one thing about WLS, is that it really has nothing to do with SiteFinder, per se. So can we table this particular discussion to some other point? Because we're really talking about wildcards here and not SiteFinder.

Steve Crocker: fair enough. Marilyn Cade.

Marilyn Cade: my name is Marilyn Cade. I work for AT&T. And as many of you know I am an elected Business Constituency Representative to the GNSO Council.

And I'd like to just make a couple of comments and start out by thanking both ICANN and the Security and Stability Advisory Committee and everyone who came here today and who is paying attention to this for being willing to work together in what I hope will, from this point forward, be the collaborative consensus-based approach that we have all worked together historically and try to work together to address these new challenges.

I think the commonality that all of us have is that the vision we share of the Internet is actually quite similar. Only a few years ago, in 1997, there were less than 20,000 -- excuse me, less than 20 million hosts and less than 50 million users. Today there are well over half a trillion users and about 180 million hosts. And that is growing.

As John Klensin said, increasingly, people in other countries with other languages want to use those languages in order to communicate over the Internet. So I think the vision that the Internet is important is a shared vision.

Here's my concern: governments are looking very hard to see whether or not the consensus-based approach can work, whether the private sector can be trusted to solve these very complicated challenges. And I would say it behooves all of us to understand that, yes, there are largely technical problems and questions that have to be addressed and that stability is in fact more than just those hard technical questions.

And ICANN will have a process that I call the parking lot where those other questions had to be parked today. And we are going to have to address those questions in the very near term. So, going forward, I guess my request to the Security and Stability Advisory Committee is, those of us who are going to be trying to answer some of those other questions are going to be very dependent on you for close consultation and for your advice.

Robert Enger: Hi, I'm Robert Enger.

There seems to be a large amount of money to be made here. We will, with the Verisign system coming online, it will essentially take money out of the pockets of Microsoft. They may wish to respond. They may make changes to Internet Explorer that will try and recapture some of those typo hits in order to get their revenue restored. This may lead Verisign to make changes, if that is doable, that would be a counter measure. This would only impact these two companies initially as they duel for the availability typos and therefore the advertising revenue.

But to the extent Verisign makes changes, whatever changes are feasible, this may disenfranchise these third parties who have installed whatever work around there are. So while the initial loss would seem to be only to Verisign and Microsoft, it may have the ancillary effect of disadvantaging third parties who are also trying to work around.

Steve Crocker: if I understand what you're saying, you have no desire to get in between Verisign and Microsoft, and neither do I. You just don't want to be too close to the action.

Robert Enger: there may be a continuing round of measure and counter measure in order to maintain their key clicks and therefore the revenue stream. That may also disenfranchise others.

Howard Berkowitz: my question is a stability one based on several assumptions.

Let us assume that an evolved SiteFinder, whether we have changes in application protocols for a new error message, so, essentially, the question of can it respond to any new protocol with a (inaudible) response is solved for the SiteFinder code and SiteFinder works.

Now, let's take this in a different standpoint now. We have a limited number, that seems to be sufficient, of F root servers. They're geographically dispersed and so on. It's a small enough set that we can have a table of the addresses of those root servers. The question comes up is, though, that those aren't queried relatively often.

Potentially, SiteFinder usage would go up not by the number of DNS resolvers or DNS (inaudible) caches, domains, whatever; it would go up by the number of user sessions. That would require, presumably, more SiteFinder instantiations. In order to achieve stability , how many SiteFinder implementations would we need? What would we -- let's say this becomes part of the infrastructure. How many do we have to have? How do we identify that it may be an any cast that we get to it so we're not sure about the address, or do we designate a block of address that these are SiteFinder-returned or so on?

In other words, if this were adopt as a large scale, it seems like -- well, obviously, we can have just one SiteFinder cluster, that's a single point of failure. How do you bring it up to the level of robustness that we have in BGP?

Steve Crocker: let me help out my good friends at Verisign. That's the kind of question that I call a Brair Rabbit question.

Please don't throw me in that Briar patch.

Howard Berkowitz: I'm trying to get this constructively.

Steve Crocker: let me try and then you get a second shot for free.

I think one of Verisign's demonstrated long suits is the ability to run robust services, replicated, hardened, that have capacity and withstand load changes over time, have geographic distribution, have multiple points of interconnection, and so forth. And I think that, you know, there's sort of demonstrated capacity there.

So if it's just a question of how do they handle the load that SiteFinder might be, I think that that would be a delicious question to them. They'd be -- you know, if that's the only issue, they'd be in good shape for that.

Howard Berkowitz: the other issues to consider would be, is, one, if someone other than Verisign gets -- let's -- we have the generic SiteFinder function, and through the course of business, someone else gets it or another domain does the same thing.

Will -- you know, how will they deal with it? Will other top-level domain operators have the same capability? And then in the real world, what if this becomes -- comes under deliberate attack? What -- in other words, what are the engineering requirements that we can look toward? if SiteFinder-like functions are used, how can we -- if it becomes as critical to infrastructure as does BGP routing, what are the requirements to make it dependable?

And I think that needs to be not just a Verisign business decision, but a stability point that's put out in general for the communities to say, yeah, that's stability. Because we do that for routing.

Steve Crocker: I'll pass this down the line in a minute. But I meant to add previously, in comparison -- comparison with BGP opens an entirely different can of worms and that is not something that I would take as a gold standard.

Howard Berkowitz: of course. Would you accept at least brightly-shined copper?

Steve Crocker: all that glitters is not gold. So I don't know where we go.

(laughter.)

Steven Bellovin: one store to -- never mind. I mostly agree with you, Steve.

I think that there's ample examples in the industry, not just Verisign, but the industry as a whole, of how to replicate services to avoid single points of failure under normal conditions. I share your concern about abnormal conditions, in particular, as a security guy, about some group of no-goodnics deciding to launch a DDoS attack on this service.

And the question becomes, you know, can this be replicated with a content distribution network far enough? You know, we saw that when the worm de jure was threatening to take out Windows Update, well, Microsoft was able to get that distributed well enough, aided by some infelllessities in the worm. Maybe we won't be so lucky next time.

But the stability of this under attack is an interesting question. Most of the attacks haven't lasted for that long, and this is -- again, this is an error condition, not as if -- it's not like taking out the DNS servers. It's like taking out the error message. Not good, but not as bad.

Ken Silva: I agree with a lot of what Steve just said here. First of all, I will -- I will tell you that most of what you suggested could happen already has happened. Okay? I think, you know, I will just say this, that I think our track record of running systems at 100% or near 100% availability for the last half dozen years I think speaks for itself. I think all of those things were taken into consideration when the SiteFinder architecture was put together, when the system was deployed. And every minute of every day as long as it's been up.

Howard Berkowitz: may I again ask you, meaning to be constructively, would it be possible to put out for any new TLD that you -- assuming they use the same technology, what are the operational requirements that should be expected of a registry offering such a service to give such ability?

I mean, let's grant you know how to do it, but, you know, the full registry gets created. Are they, essentially, a weak one? And I think it would be good to get those availability concepts out into the public.

Ken Silva: absolutely. I think that's a dynamite suggestion. And I exactly think that that's something that should be documented and published. And, in fact, part of what you suggest, not all of what you just suggested, but part of that, has been specifically addressed and documented in the white paper that's on our web site now for how we believe the wildcard should respond to different types of applications and different types of scenarios. So that's about half, I think, of what you asked for. I think the other half is a worthwhile goal that I think the community should take on.

Steve Crocker: thank you.

Ah.

Chuck Gomes: my apologies for taking so much time. Chuck Gomes again from Verisign.

First of all, I want to just make sure everybody in the audience -- I know everybody up here understands this very well. Matt Larson tried to correct this earlier today. We're not talking about root servers here with regard to the SiteFinder service. We're talking about the com/net constellation of servers. And I want to make sure nobody thinks that we're talking about the root servers when we leave here today. And I understand why the terms are interchanged from history and so forth. So I just want to make sure of that.

Two things: number one, you know, we just posted today, as was stated earlier today, a thorough response to the IAB concerns. And that is available on our site. We sent that to ICANN. I'm sure it'll be available on the ICANN site as well. And encourage people to look at that and certainly use the e-mail address that we gave so that we have feedback. Specific feedback would be very much appreciated.

One parting question is, are we talking about wildcards based on a standard or are we talking about change made to the DNS?

Steve Crocker: any other comments, issues? We're in the broad, open area, and very rapidly we'll move into closing remarks.

Robert Seastrom: Rob Seastrom again.

I'd like to put forth a personal opinion here, which is that both of these -- we talk about making wildcards in any zone where there's sub-delegation of that zone to other people. And in a greater sense, even whenever there's a customer/vendor relationship there, that extreme caution is necessary and that it is, in all probability, violating two principles which have served us well for interoperability in years past. And those being the conservative in what you send and liberal in what you accept and the principle of least user astonishment.

That's all.

Steve Crocker: thank you.

All right.

Well, I think -- oh.

K.C. Claffy: not uh-oh. K.C. Claffy -- from the Cooperative Association for Internet Data Analysis.

So we're a research group in San Diego and we're really big fans of real data , as hard as it is to get. So I'm fairly empathetic to Verisign's continued request for hard data backing up some of the issues articulated here. What I'm a little less comfortable with is, you said that you have partners that you've worked with to do pretesting on this SiteFinder. You haven't named them yet and haven't been asked to. So maybe you could name them. And keep in mind I'm a researcher. So I like things to be rigorous and published and open.

I'm wondering if you would be willing to make available the results of that research, how you went about doing the experimental design to prove that in fact there were no effects or you weren't anticipating seeing any effects. Because we have, obviously, seen a lot of effects. Personally, my spam more than doubled the week this happened. I didn't do an experimental design myself to figure out what are the causes of that. There are many causes of spam increasing in any given week. But I would be really intrigued at how you approached the testing of this particular change.

An interesting anecdote you might want to check out, CAIDA gave a talk and did a study a year ago on some attribute of Microsoft software that related to DHCP and updating DNS with DHCP mappings automatically , which resulted, because of a combination of different attributes, none by itself which were wrong, but taken together in the system, ended up causing a ton of traffic to the root name servers that shouldn't be there.

And this is something that Microsoft didn't know; it didn't know it was happening until we pointed it out. We analyzed the root name servers. Paul actually asked us to analyze F root data. So they didn't know about it until it sort of hit the core of the system.

I think there's a lot of things that are like this now in the Internet because you have so many little things that work together and it's very hard to test the space in between without launching it. So I'm very interested in when you say, "we tested this with partners," what that means in really great detail, if you can make that public, I think it would go a long way.

Steve Crocker: thank you.

Let me comment that KC at CAIDA has done stellar work in measurement and has had her fingers dirty with real data for years. And I think has been a real pioneer in bringing strong involvement with serious measurement into the network. And there is the gold standard of measurement activities.

Thanks, KC.

All right. Well --

Steven Bellovin: I was going to ask, KC, if you've done any measurements of this over the last few weeks.

K.C. Claffy: oh, that reminds me of another comment I didn't make. It costs a lot of money to do this kind of research, okay?

(laughter.)

K.C. Claffy: by the way, Verisign keeps asking for hard data. And one of the guys asked for how much money it cost. We don't have the resources to drop everything and do this. We're trying to. But I doesn't have hard data at the moment.

Steve Crocker: thank you.

Any -- any other comments from panel members?

Mike St. John: are we going to get a response to her question?

Steve Crocker: that's up to Verisign.

>>: what was it?

Steve Crocker: is Verisign going to respond to the question of -- making the data available and the methodology behind it.

I think -- is that the question?

Ben Turner: as we've offered to the committee before, our plan is to make available all the data that we have so that everybody can see the same results that we've evaluated and have an open evaluation of that data.

Steve Crocker: good.

Well, we look forward to that.

K.C. Claffy: I love the data you have. But I'm, in particular, interested in what you did before that convinced you that this was an okay thing to do. Because that's what I think the community is kind of freaking out about. Oh, my god, they did this arbitrary thing, didn't check with anyone, didn't work with ICANN. They decided it was okay, all right, fine, I'm not ready to freak out about that.

But how was the decision made? And all the data that you've gathered afterwards, of course, I'm interested in as well.

Ben Turner: so we're happy to sell it to you if you can get a grant.

(laughter.)

Ben Turner: no, we plan to make all of our data available.

So... --

K.C. Claffy: actually you're going to have to pay me to take it.

(applause.)

Steve Crocker: boy, I don't know how to follow that.

(laughter.)

Steve Crocker: I think that's a perfect note to bring this to a close. As we mentioned earlier, this has been part of a process. It was preceded by very vigorous acquisition of inputs, which is still ongoing.

The main purpose of having the meeting today was to force-feed that process and bring the information that we had gathered into view. It is not a decision-making event today, but one, as I say, of information-gathering and presentation.

We are going to have a follow-up meeting to open up some of these questions in more detail. We'll prepare the information that we have and subject it to the usual close scrutiny internally to see that we've not overreached and that we are as accurate as we know how to be. And then we'll present the material to both the external community, the public, and the technical community, and, of course, to the other parts of ICANN for the decision processes and negotiation processes and consensus processes and all of the other processes that we've come to know and love within ICANN.

I thank everybody for coming. This meeting was put together on very short notice. It was stressful on all of us who participated. And I hope it's been useful to everyone who has come and taken their time.

Again, I thank the staff at CSIS. I thank the volunteers who have helped put the meeting together. Our committee, of course, and support from a number of places.

The ICANN staff worked very well. We have had an amazing amount of support almost overnight put together fairly complicated logistics. We have a quite excellent crew over here. I feel like I'm saluting the band and we should all sort of clap for the band.

(applause.)

Steve Crocker: and if I haven't forgotten anything, -- oops, maybe I have.

Ah, let people know that the -- as rapidly as we can acquire them from each of the speakers, and I think we're partway there, the slides will be posted. We put together a focused web site for this -- not only for this meeting, but for this sort of sequence, secsac.icann.org,

secsac.icann.org.

Most of you are probably familiar with it already because that's how you registered if you did register. And so in the famous words, simply watch that space, and you'll see the subsequent developments.

Again, thank you all for coming.

And this brings this meeting to a close.

(applause.)

(meeting concluded.)

Comments concerning the layout, construction and functionality of this site
should be sent to webmaster@icann.org.

Page Updated 10-Oct-2003
© 2003 The Internet Corporation for Assigned Names and Numbers. All rights reserved.