How Webs Servers Work

How Web Servers Work

by Marshall Brain

Photo courtesy Shopping.com IBM Netfinity 5500 server

Have you ever wondered about the mechanisms that delivered this page to you? Chances are you are sitting at a computer right now, viewing this page in a browser. So, when you clicked on the link for this page, or typed in its URL (uniform resource locator), what happened behind the scenes to bring this page onto your screen?

If you've ever been curious about the process, or have ever wanted to know some of the specific mechanisms that allow you to surf the Internet, then read on. In this article, you will learn how Web servers bring pages into your home, school or office. Let's get started!

The Basic Process
Let's say that you are sitting at your computer, surfing the Web, and you get a call from a friend who says, "I just read a great article! Type in this URL and check it out. It's at http://computer.howstuffworks.com/web-server.htm." So you type that URL into your browser and press return. And magically, no matter where in the world that URL lives, the page pops up on your screen.

At the most basic level possible, the following diagram shows the steps that brought that page to your screen:

Your browser formed a connection to a Web server, requested a page and received it.

On the next page, we'll dig a bit deeper.

Behind the Scenes
If you want to get into a bit more detail on the process of getting a Web page onto your computer screen, here are the basic steps that occurred behind the scenes:

• The browser broke the URL into three parts:
  1. The protocol ("http")
  2. The server name ("www.howstuffworks.com")
  3. The file name ("web-server.htm")

• The browser communicated with a name server to translate the server name "www.howstuffworks.com" into an IP Address, which it uses to connect to the server machine.

• The browser then formed a connection to the server at that IP address on port 80. (We'll discuss ports later in this article.)

• Following the HTTP protocol, the browser sent a GET request to the server, asking for the file "http://computer.howstuffworks.com/web-server.htm." (Note that cookies may be sent from browser to server with the GET request -- see How Internet Cookies Work for details.)

• The server then sent the HTML text for the Web page to the browser. (Cookies may also be sent from server to browser in the header for the page.)

• The browser read the HTML tags and formatted the page onto your screen.

If you've never explored this process before, that's a lot of new vocabulary. To understand this whole process in detail, you need to learn about IP addresses, ports, protocols... The following sections will lead you through a complete explanation.

The Internet
So what is "the Internet"? The Internet is a gigantic collection of millions of computers, all linked together on a computer network. The network allows all of the computers to communicate with one another. A home computer may be linked to the Internet using a phone-line modem, DSL or cable modem that talks to an Internet service provider (ISP). A computer in a business or university will usually have a network interface card (NIC) that directly connects it to a local area network (LAN) inside the business. The business can then connect its LAN to an ISP using a high-speed phone line like a T1 line. A T1 line can handle approximately 1.5 million bits per second, while a normal phone line using a modem can typically handle 30,000 to 50,000 bits per second.

ISPs then connect to larger ISPs, and the largest ISPs maintain fiber-optic "backbones" for an entire nation or region. Backbones around the world are connected through fiber-optic lines, undersea cables or satellite links (see An Atlas of Cyberspaces for some interesting backbone maps). In this way, every computer on the Internet is connected to every other computer on the Internet.

Clients and Servers
In general, all of the machines on the Internet can be categorized as two types: servers and clients. Those machines that provide services (like Web servers or FTP servers) to other machines are servers. And the machines that are used to connect to those services are clients. When you connect to Yahoo! at www.yahoo.com to read a page, Yahoo! is providing a machine (probably a cluster of very large machines), for use on the Internet, to service your request. Yahoo! is providing a server. Your machine, on the other hand, is probably providing no services to anyone else on the Internet. Therefore, it is a user machine, also known as a client. It is possible and common for a machine to be both a server and a client, but for our purposes here you can think of most machines as one or the other.

A server machine may provide one or more services on the Internet. For example, a server machine might have software running on it that allows it to act as a Web server, an e-mail server and an FTP server. Clients that come to a server machine do so with a specific intent, so clients direct their requests to a specific software server running on the overall server machine. For example, if you are running a Web browser on your machine, it will most likely want to talk to the Web server on the server machine. Your Telnet application will want to talk to the Telnet server, your e-mail application will talk to the e-mail server, and so on...

IP Addresses
To keep all of these machines straight, each machine on the Internet is assigned a unique address called an IP address. IP stands for Internet protocol, and these addresses are 32-bit numbers, normally expressed as four "octets" in a "dotted decimal number." A typical IP address looks like this:

          216.27.61.137

The four numbers in an IP address are called octets because they can have values between 0 and 255, which is 2⁸ possibilities per octet.

Every machine on the Internet has a unique IP address. A server has a static IP address that does not change very often. A home machine that is dialing up through a modem often has an IP address that is assigned by the ISP when the machine dials in. That IP address is unique for that session -- it may be different the next time the machine dials in. This way, an ISP only needs one IP address for each modem it supports, rather than for each customer.

If you are working on a Windows machine, you can view a lot of the Internet information for your machine, including your current IP address and hostname, with the command WINIPCFG.EXE (IPCONFIG.EXE for Windows 2000/XP). On a UNIX machine, type nslookup at the command prompt, along with a machine name, like www.howstuffworks.com -- e.g. "nslookup www.howstuffworks.com" -- to display the IP address of the machine, and you can use the command hostname to learn the name of your machine. (For more information on IP addresses, see IANA.)

As far as the Internet's machines are concerned, an IP address is all you need to talk to a server. For example, in your browser, you can type the URL http://209.116.69.66 and arrive at the machine that contains the Web server for HowStuffWorks. On some servers, the IP address alone is not sufficient, but on most large servers it is -- keep reading for details.

Domain Names
Because most people have trouble remembering the strings of numbers that make up IP addresses, and because IP addresses sometimes need to change, all servers on the Internet also have human-readable names, called domain names. For example, www.howstuffworks.com is a permanent, human-readable name. It is easier for most of us to remember www.howstuffworks.com than it is to remember 209.116.69.66.

The name www.howstuffworks.com actually has three parts:

1. The host name ("www")
2. The domain name ("howstuffworks")
3. The top-level domain name ("com")

Domain names within the ".com" domain are managed by the registrar called VeriSign. VeriSign also manages ".net" domain names. Other registrars (like RegistryPro, NeuLevel and Public Interest Registry) manage the other domains (like .pro, .biz and .org). VeriSign creates the top-level domain names and guarantees that all names within a top-level domain are unique. VeriSign also maintains contact information for each site and runs the "whois" database. The host name is created by the company hosting the domain. "www" is a very common host name, but many places now either omit it or replace it with a different host name that indicates a specific area of the site. For example, in encarta.msn.com, the domain name for Microsoft's Encarta encyclopedia, "encarta" is designated as the host name instead of "www."

Name Servers

The whois Command On a UNIX machine, you can use the whois command to look up information about a domain name. You can do the same thing using the whois form at VeriSign. If you type in a domain name, like "howstuffworks.com," it will return to you the registration information for that domain, including its IP address.

A set of servers called domain name servers (DNS) maps the human-readable names to the IP addresses. These servers are simple databases that map names to IP addresses, and they are distributed all over the Internet. Most individual companies, ISPs and universities maintain small name servers to map host names to IP addresses. There are also central name servers that use data supplied by VeriSign to map domain names to IP addresses.

If you type the URL "http://computer.howstuffworks.com/web-server.htm" into your browser, your browser extracts the name "www.howstuffworks.com," passes it to a domain name server, and the domain name server returns the correct IP address for www.howstuffworks.com. A number of name servers may be involved to get the right IP address. For example, in the case of www.howstuffworks.com, the name server for the "com" top-level domain will know the IP address for the name server that knows host names, and a separate query to that name server, operated by the HowStuffWorks ISP, may deliver the actual IP address for the HowStuffWorks server machine.

On a UNIX machine, you can access the same service using the nslookup command. Simply type a name like "www.howstuffworks.com" into the command line, and the command will query the name servers and deliver the corresponding IP address to you.

So here it is: The Internet is made up of millions of machines, each with a unique IP address. Many of these machines are server machines, meaning that they provide services to other machines on the Internet. You have heard of many of these servers: e-mail servers, Web servers, FTP servers, Gopher servers and Telnet servers, to name a few. All of these are provided by server machines.

Ports
Any server machine makes its services available to the Internet using numbered ports, one for each service that is available on the server. For example, if a server machine is running a Web server and an FTP server, the Web server would typically be available on port 80, and the FTP server would be available on port 21. Clients connect to a service at a specific IP address and on a specific port.

Each of the most well-known services is available at a well-known port number. Here are some common port numbers:

• echo 7
• daytime 13
• qotd 17 (Quote of the Day)
• ftp 21
• telnet 23
• smtp 25 (Simple Mail Transfer, meaning e-mail)
• time 37
• nameserver 53
• nicname 43 (Who Is)
• gopher 70
• finger 79
• WWW 80

If the server machine accepts connections on a port from the outside world, and if a firewall is not protecting the port, you can connect to the port from anywhere on the Internet and use the service. Note that there is nothing that forces, for example, a Web server to be on port 80. If you were to set up your own machine and load Web server software on it, you could put the Web server on port 918, or any other unused port, if you wanted to. Then, if your machine were known as xxx.yyy.com, someone on the Internet could connect to your server with the URL http://xxx.yyy.com:918. The ":918" explicitly specifies the port number, and would have to be included for someone to reach your server. When no port is specified, the browser simply assumes that the server is using the well-known port 80.

Protocols
Once a client has connected to a service on a particular port, it accesses the service using a specific protocol. The protocol is the pre-defined way that someone who wants to use a service talks with that service. The "someone" could be a person, but more often it is a computer program like a Web browser. Protocols are often text, and simply describe how the client and server will have their conversation.

Perhaps the simplest protocol is the daytime protocol. If you connect to port 13 on a machine that supports a daytime server, the server will send you its impression of the current date and time and then close the connection. The protocol is, "If you connect to me, I will send you the date and time and then disconnect." Most UNIX machines support this server. If you would like to try it out, you can connect to one with the Telnet application. In UNIX, the session would look like this:

%telnet web67.ntx.net 13

Trying 216.27.61.137...

Connected to web67.ntx.net.

Escape character is '^]'.

Sun Oct 25 08:34:06 1998

Connection closed by foreign host.

On a Windows machine, you can access this server by typing "telnet web67.ntx.net 13" at the MSDOS prompt.

In this example, web67.ntx.net is the server's UNIX machine, and 13 is the port number for the daytime service. The Telnet application connects to port 13 (telnet naturally connects to port 23, but you can direct it to connect to any port), then the server sends the date and time and disconnects. Most versions of Telnet allow you to specify a port number, so you can try this using whatever version of Telnet you have available on your machine.

Most protocols are more involved than daytime and are specified in Request for Comment (RFC) documents that are publicly available (see http://sunsite.auc.dk/RFC/ for a nice archive of all RFCs). Every Web server on the Internet conforms to the HTTP protocol, summarized nicely in The Original HTTP as defined in 1991. The most basic form of the protocol understood by an HTTP server involves just one command: GET. If you connect to a server that understands the HTTP protocol and tell it to "GET filename," the server will respond by sending you the contents of the named file and then disconnecting. Here's a typical session:

%telnet www.howstuffworks.com 80

Trying 216.27.61.137...

Connected to howstuffworks.com.

Escape character is '^]'.

GET http://computer.howstuffworks.com/



Connection closed by foreign host.

In the original HTTP protocol, all you would have sent was the actual filename, such as "/" or "/web-server.htm." The protocol was later modified to handle the sending of the complete URL. This has allowed companies that host virtual domains, where many domains live on a single machine, to use one IP address for all of the domains they host. It turns out that hundreds of domains are hosted on 209.116.69.66 -- the HowStuffWorks IP address.

Putting It All Together
Now you know a tremendous amount about the Internet. You know that when you type a URL into a browser, the following steps occur:

• The browser breaks the URL into three parts:
  1. The protocol ("http")
  2. The server name ("www.howstuffworks.com")
  3. The file name ("web-server.htm")

• The browser communicates with a name server to translate the server name, "www.howstuffworks.com," into an IP address, which it uses to connect to that server machine.

• The browser then forms a connection to the Web server at that IP address on port 80.

• Following the HTTP protocol, the browser sends a GET request to the server, asking for the file "http://computer.howstuffworks.com/web-server.htm." (Note that cookies may be sent from browser to server with the GET request -- see How Internet Cookies Work for details.)

• The server sends the HTML text for the Web page to the browser. (Cookies may also be sent from server to browser in the header for the page.)

• The browser reads the HTML tags and formats the page onto your screen.

Extras: Security
You can see from this description that a Web server can be a pretty simple piece of software. It takes the file name sent in with the GET command, retrieves that file and sends it down the wire to the browser. Even if you take into account all of the code to handle the ports and port connections, you could easily create a C program that implements a simple Web server in less than 500 lines of code. Obviously, a full-blown enterprise-level Web server is more involved, but the basics are very simple.

Most servers add some level of security to the serving process. For example, if you have ever gone to a Web page and had the browser pop up a dialog box asking for your name and password, you have encountered a password-protected page. The server lets the owner of the page maintain a list of names and passwords for those people who are allowed to access the page; the server lets only those people who know the proper password see the page. More advanced servers add further security to allow an encrypted connection between server and browser, so that sensitive information like credit card numbers can be sent on the Internet.

That's really all there is to a Web server that delivers standard, static pages. Static pages are those that do not change unless the creator edits the page.

Extras: Dynamic Pages
But what about the Web pages that are dynamic? For example:

• Any guest book allows you to enter a message in an HTML form, and the next time the guest book is viewed, the page will contain the new entry.

• The whois form at Network Solutions allows you to enter a domain name on a form, and the page returned is different depending on the domain name entered.

• Any search engine lets you enter keywords on an HTML form, and then it dynamically creates a page based on the keywords you enter.

In all of these cases, the Web server is not simply "looking up a file." It is actually processing information and generating a page based on the specifics of the query. In almost all cases, the Web server is using something called CGI scripts to accomplish this feat. CGI scripts are a topic unto themselves, and are described in the HowStuffWorks article How CGI Scripting Work.

For more information on Web servers and related topics, check out the links on the next page.

Lots More Information

Related HowStuffWorks Articles

• How Internet Infrastructure Works
• How Domain Name Servers Work
• How ASPs Work
• How E-mail Works
• How File Sharing Works
• How Internet Cookies Work
• How Network Address Translation Works
• How CGI Scripting Works
• Where are all the Internet domain names registered and maintained?
• Why do some Web sites include "www" in the URL while others don't?

More Great Links

• Whatis.com - great dictionary of Internet terms
• The Original HTTP as defined in 1991
• Firewalls FAQ
• IANA
• Network Solutions
• RFC 1340 - Assigned numbers
• Setting up your own DNS
• Web Security
• HTTP overview

How Internet Infrastructure Works

How Internet Infrastructure Works

by Jeff Tyson

One of the greatest things about the Internet is that nobody really owns it. It is a global collection of networks, both big and small. These networks connect together in many different ways to form the single entity that we know as the Internet. In fact, the very name comes from this idea of interconnected networks.

Since its beginning in 1969, the Internet has grown from four host computer systems to tens of millions. However, just because nobody owns the Internet, it doesn't mean it is not monitored and maintained in different ways. The Internet Society, a non-profit group established in 1992, oversees the formation of the policies and protocols that define how we use and interact with the Internet.

In this article, you will learn about the basic underlying structure of the Internet. You will learn about domain name servers, network access points and backbones. But first you will learn about how your computer connects to others.

A Hierarchy of Networks
Every computer that is connected to the Internet is part of a network, even the one in your home. For example, you may use a modem and dial a local number to connect to an Internet Service Provider (ISP). At work, you may be part of a local area network (LAN), but you most likely still connect to the Internet using an ISP that your company has contracted with. When you connect to your ISP, you become part of their network. The ISP may then connect to a larger network and become part of their network. The Internet is simply a network of networks.

Most large communications companies have their own dedicated backbones connecting various regions. In each region, the company has a Point of Presence (POP). The POP is a place for local users to access the company's network, often through a local phone number or dedicated line. The amazing thing here is that there is no overall controlling network. Instead, there are several high-level networks connecting to each other through Network Access Points or NAPs.

When you connect to the Internet, your computer becomes part of a network.

A Network Example
Here's an example. Imagine that Company A is a large ISP. In each major city, Company A has a POP. The POP in each city is a rack full of modems that the ISP's customers dial into. Company A leases fiber optic lines from the phone company to connect the POPs together (see, for example, this UUNET Data Center Connectivity Map).

Imagine that Company B is a corporate ISP. Company B builds large buildings in major cities and corporations locate their Internet server machines in these buildings. Company B is such a large company that it runs its own fiber optic lines between its buildings so that they are all interconnected.

In this arrangement, all of Company A's customers can talk to each other, and all of Company B's customers can talk to each other, but there is no way for Company A's customers and Company B's customers to intercommunicate. Therefore, Company A and Company B both agree to connect to NAPs in various cities, and traffic between the two companies flows between the networks at the NAPs.

In the real Internet, dozens of large Internet providers interconnect at NAPs in various cities, and trillions of bytes of data flow between the individual networks at these points. The Internet is a collection of huge corporate networks that agree to all intercommunicate with each other at the NAPs. In this way, every computer on the Internet connects to every other.

Bridging The Divide
All of these networks rely on NAPs, backbones and routers to talk to each other. What is incredible about this process is that a message can leave one computer and travel halfway across the world through several different networks and arrive at another computer in a fraction of a second!

The routers determine where to send information from one computer to another. Routers are specialized computers that send your messages and those of every other Internet user speeding to their destinations along thousands of pathways. A router has two separate, but related, jobs:

• It ensures that information doesn't go where it's not needed. This is crucial for keeping large volumes of data from clogging the connections of "innocent bystanders."
• It makes sure that information does make it to the intended destination.

In performing these two jobs, a router is extremely useful in dealing with two separate computer networks. It joins the two networks, passing information from one to the other. It also protects the networks from one another, preventing the traffic on one from unnecessarily spilling over to the other. Regardless of how many networks are attached, the basic operation and function of the router remains the same. Since the Internet is one huge network made up of tens of thousands of smaller networks, its use of routers is an absolute necessity. For more information, read How Routers Work.

Backbones
The National Science Foundation (NSF) created the first high-speed backbone in 1987. Called NSFNET, it was a T1 line that connected 170 smaller networks together and operated at 1.544 Mbps (million bits per second). IBM, MCI and Merit worked with NSF to create the backbone and developed a T3 (45 Mbps) backbone the following year.

Backbones are typically fiber optic trunk lines. The trunk line has multiple fiber optic cables combined together to increase the capacity. Fiber optic cables are designated OC for optical carrier, such as OC-3, OC-12 or OC-48. An OC-3 line is capable of transmitting 155 Mbps while an OC-48 can transmit 2,488 Mbps (2.488 Gbps). Compare that to a typical 56K modem transmitting 56,000 bps and you see just how fast a modern backbone is.

Today there are many companies that operate their own high-capacity backbones, and all of them interconnect at various NAPs around the world. In this way, everyone on the Internet, no matter where they are and what company they use, is able to talk to everyone else on the planet. The entire Internet is a gigantic, sprawling agreement between companies to intercommunicate freely.

Internet Protocol: IP Addresses
Every machine on the Internet has a unique identifying number, called an IP Address. The IP stands for Internet Protocol, which is the language that computers use to communicate over the Internet. A protocol is the pre-defined way that someone who wants to use a service talks with that service. The "someone" could be a person, but more often it is a computer program like a Web browser.

A typical IP address looks like this:

216.27.61.137

To make it easier for us humans to remember, IP addresses are normally expressed in decimal format as a dotted decimal number like the one above. But computers communicate in binary form. Look at the same IP address in binary:

11011000.00011011.00111101.10001001

The four numbers in an IP address are called octets, because they each have eight positions when viewed in binary form. If you add all the positions together, you get 32, which is why IP addresses are considered 32-bit numbers. Since each of the eight positions can have two different states (1 or zero), the total number of possible combinations per octet is 2⁸ or 256. So each octet can contain any value between zero and 255. Combine the four octets and you get 2³² or a possible 4,294,967,296 unique values!

Out of the almost 4.3 billion possible combinations, certain values are restricted from use as typical IP addresses. For example, the IP address 0.0.0.0 is reserved for the default network and the address 255.255.255.255 is used for broadcasts.

The octets serve a purpose other than simply separating the numbers. They are used to create classes of IP addresses that can be assigned to a particular business, government or other entity based on size and need. The octets are split into two sections: Net and Host. The Net section always contains the first octet. It is used to identify the network that a computer belongs to. Host (sometimes referred to as Node) identifies the actual computer on the network. The Host section always contains the last octet. There are five IP classes plus certain special addresses. You can learn more about IP classes at What is an IP address?.

Internet Protocol: Domain Name System
When the Internet was in its infancy, it consisted of a small number of computers hooked together with modems and telephone lines. You could only make connections by providing the IP address of the computer you wanted to establish a link with. For example, a typical IP address might be 216.27.22.162. This was fine when there were only a few hosts out there, but it became unwieldy as more and more systems came online.

The first solution to the problem was a simple text file maintained by the Network Information Center that mapped names to IP addresses. Soon this text file became so large it was too cumbersome to manage. In 1983, the University of Wisconsin created the Domain Name System (DNS), which maps text names to IP addresses automatically. This way you only need to remember www.howstuffworks.com, for example, instead of HowStuffWorks.com's IP address.

Uniform Resource Locators
When you use the Web or send an e-mail message, you use a domain name to do it. For example, the Uniform Resource Locator (URL) "http://www.howstuffworks.com" contains the domain name howstuffworks.com. So does this e-mail address: example@howstuffworks.com. Every time you use a domain name, you use the Internet's DNS servers to translate the human-readable domain name into the machine-readable IP address. Check out How Domain Name Servers Work for more in-depth information on DNS.

Top-level domain names, also called first-level domain names, include .COM, .ORG, .NET, .EDU and .GOV. Within every top-level domain there is a huge list of second-level domains. For example, in the .COM first-level domain there is:

• HowStuffWorks
• Yahoo
• Microsoft

Every name in the .COM top-level domain must be unique. The left-most word, like www, is the host name. It specifies the name of a specific machine (with a specific IP address) in a domain. A given domain can, potentially, contain millions of host names as long as they are all unique within that domain.

DNS servers accept requests from programs and other name servers to convert domain names into IP addresses. When a request comes in, the DNS server can do one of four things with it:

1. It can answer the request with an IP address because it already knows the IP address for the requested domain.
2. It can contact another DNS server and try to find the IP address for the name requested. It may have to do this multiple times.
3. It can say, "I don't know the IP address for the domain you requested, but here's the IP address for a DNS server that knows more than I do."
4. It can return an error message because the requested domain name is invalid or does not exist.

A DNS Example
Let's say that you type the URL www.howstuffworks.com into your browser. The browser contacts a DNS server to get the IP address. A DNS server would start its search for an IP address by contacting one of the root DNS servers. The root servers know the IP addresses for all of the DNS servers that handle the top-level domains (.COM, .NET, .ORG, etc.). Your DNS server would ask the root for www.howstuffworks.com, and the root would say, "I don't know the IP address for www.howstuffworks.com, but here's the IP address for the .COM DNS server."

Your name server then sends a query to the .COM DNS server asking it if it knows the IP address for www.howstuffworks.com. The DNS server for the COM domain knows the IP addresses for the name servers handling the www.howstuffworks.com domain, so it returns those.

Your name server then contacts the DNS server for www.howstuffworks.com and asks if it knows the IP address for www.howstuffworks.com. It actually does, so it returns the IP address to your DNS server, which returns it to the browser, which can then contact the server for www.howstuffworks.com to get a Web page.

One of the keys to making this work is redundancy. There are multiple DNS servers at every level, so that if one fails, there are others to handle the requests. The other key is caching. Once a DNS server resolves a request, it caches the IP address it receives. Once it has made a request to a root DNS server for any .COM domain, it knows the IP address for a DNS server handling the .COM domain, so it doesn't have to bug the root DNS servers again for that information. DNS servers can do this for every request, and this caching helps to keep things from bogging down.

Even though it is totally invisible, DNS servers handle billions of requests every day and they are essential to the Internet's smooth functioning. The fact that this distributed database works so well and so invisibly day in and day out is a testimony to the design. Be sure to read How Domain Name Servers Work for more information on DNS.

Clients and Servers
Internet servers make the Internet possible. All of the machines on the Internet are either servers or clients. The machines that provide services to other machines are servers. And the machines that are used to connect to those services are clients. There are Web servers, e-mail servers, FTP servers and so on serving the needs of Internet users all over the world.

When you connect to www.howstuffworks.com to read a page, you are a user sitting at a client's machine. You are accessing the HowStuffWorks Web server. The server machine finds the page you requested and sends it to you. Clients that come to a server machine do so with a specific intent, so clients direct their requests to a specific software server running on the server machine. For example, if you are running a Web browser on your machine, it will want to talk to the Web server on the server machine, not the e-mail server.

A server has a static IP address that does not change very often. A home machine that is dialing up through a modem, on the other hand, typically has an IP address assigned by the ISP every time you dial in. That IP address is unique for your session -- it may be different the next time you dial in. This way, an ISP only needs one IP address for each modem it supports, rather than one for each customer.

Ports
Any server machine makes its services available using numbered ports -- one for each service that is available on the server. For example, if a server machine is running a Web server and a file transfer protocol (FTP) server, the Web server would typically be available on port 80, and the FTP server would be available on port 21. Clients connect to a service at a specific IP address and on a specific port number.

Once a client has connected to a service on a particular port, it accesses the service using a specific protocol. Protocols are often text and simply describe how the client and server will have their conversation. Every Web server on the Internet conforms to the hypertext transfer protocol (HTTP). You can learn more about Internet servers, ports and protocols by reading How Web Servers Work.

Networks, routers, NAPs, ISPs, DNS and powerful servers all make the Internet possible. It is truly amazing when you realize that all this information is sent around the world in a matter of milliseconds! The components are extremely important in modern life -- without them, there would be no Internet. And without the Internet, life would be very different indeed for many of us.

For more information on the structure of the Internet and related topics, check out the links on the next page.

Lots More Information

Related HowStuffWorks Articles

• How Domain Name Servers Work
• How Bits and Bytes Work
• How Web Servers Work
• How Routers Work
• How Routing Algorithms Work
• How LAN Switches Work
• How Ethernet Works
• How CGI Scripts Work
• How Firewalls Work
• How Home Networking Works
• What are IP Classes?
• What is a packet?
• Where are all the Internet domain names registered and maintained?

More Great Links

• NetworkComputing.com: All About IP Addresses
• BIND - the software that implements most name servers
• Domain name registries around the world
• WhatIs.com Internet Terms
• The Original HTTP as defined in 1991
• Colby.edu: The Anatomy of a URL
• IANA
• Network Solutions
• University of Toronto: The HTTP Protocol
• Ohio State: RFC 1340 - Assigned numbers
• GaryKessler.net: Setting up your own DNS
• Yahoo! Directory: Security

Search Engines

Search engine

From Wikipedia, the free encyclopedia.

(Redirected from Search engines)

The phenomenal success of the Google search engine was mainly due to its powerful Pagerank algorithm and its simple, easy-to-use interface.

A search engine is a program designed to help find files stored on a computer, for example a public server on the World Wide Web, or one's own computer. The search engine allows one to ask for media content meeting specific criteria (typically those containing a given word or phrase) and retrieving a list of files that match those criteria. A search engine often uses a previously made, and regularly updated index to look for files after the user has entered search criteria.

In the context of the Internet, search engines usually refer to the World Wide Web and not other protocols or areas. Furthermore search engines mine data available in newsgroups, large databases, or open directories like DMOZ.org. Because the data collection is automated, they are distinguished from Web directories, which are maintained by people.

The vast majority of search engine are run by private companies using proprietary algorithms and closed databases, the most popular currently being Google (with MSN Search and Yahoo! closely behind). There have been several attempts to create open-source search engines, among which are Htdig, Nutch, Egothor and OpenFTS. [1] (http://www.searchtools.com/tools/tools-opensource.html)

[edit]

How search engines work

Web search engines work by storing information about a large number of web pages, which they retrieve from the WWW itself. These pages are retrieved by a web crawler (sometimes also known as a spider) — an automated web browser which follows every link it sees. The contents of each page are then analyzed to determine how it should be indexed (for example, words are extracted from the titles, headings, or special fields called meta tags). Data about web pages is stored in an index database for use in later queries. Some search engines, such as Google, store all or part of the source page (referred to as a cache) as well as information about the web pages.

When a user comes to the search engine and makes a query, typically by giving key words, the engine looks up the index and provides a listing of best-matching web pages according to its criteria, usually with a short summary containing the document's title and sometimes parts of the text.

There is another main type: Real-time search engines, like Orase (http://www.orase.com). Such search engines don't use an index. The information that a search engine needs is only collected if a new query is started. Compared to the index based systems of Google-like search engines this real-time system has some advantages: The information are always up-to-date, there are (almost) no dead links and less system resources are needed. (Google uses almost 100.000 computers, Orase only one.) But there are some disadvantages, too: A search needs longer to be finished, for example.

The usefulness of a search engine depends on the relevance of the results it gives back. While there may be millions of Web pages that include a particular word or phrase, some pages may be more relevant, popular, or authoritative than others. Most search engines employ methods to rank the results to provide the "best" results first. How a search engine decides which pages are the best matches, and what order the results should be shown in, varies widely from one engine to another. The methods also change over time as Internet usage changes and new techniques evolve.

Most Web search engines are commercial ventures supported by advertising revenue and, as a result, some employ the controversial practice of allowing advertisers to pay money to have their listings ranked higher in search results.

[edit]

History

The first Web search engine was "Wandex", a now-defunct index collected by the World Wide Web Wanderer, a web crawler developed by Matthew Gray at MIT in 1993. Another very early search engine, Aliweb, also appeared in 1993 and still runs today. One of the first engines to later become a major commercial endeavor was Lycos, which started at Carnegie Mellon University as a research project in 1994.

Soon after, many search engines appeared and vied for popularity. These included WebCrawler, Hotbot, Excite, Infoseek, Inktomi, and AltaVista. In some ways they competed with popular directories such as Yahoo!. Later, the directories integrated or added on search engine technology for greater functionality.

In 2002, Yahoo! acquired Inktomi and in 2003, Yahoo! acquired Overture, which owned AlltheWeb and Altavista. In 2004, Yahoo! launched its own search engine based on the combined technologies of its acquisitions and providing a service that gave pre-eminence to the Web search engine over the directory.

In December 2003, Orase published the first version of its new real-time search technology. It came with many new functions and the performance increase a lot.

Search engines were also known as some of the brightest stars in the Internet investing frenzy that occurred in the late 1990s. Several companies entered the market spectacularly, recording record gains during their initial public offerings. Some have completely taken off their public search engine, and are marketing Enterprise-only editions, such as Northern Light (http://www.northernlight.com/) which use to be part of the 8 or 9 early search engines after Lycos came out.

Before the advent of the Web, there were search engines for other protocols or uses, such as the Archie search engine for anonymous FTP sites and the Veronica search engine for the Gopher protocol.

Osmar R. Zaïane's From Resource Discovery to Knowledge Discovery on the Internet details the history of search engine technology prior to the emergence of Google.

Recent additions to the list of search engines include a9.com, AlltheWeb, Ask Jeeves, Clusty, Gigablast, Ez2Find, Teoma, WiseNut, GoHook, and Kartoo.

[edit]

Google

Around 2001, the Google search engine rose to prominence. Its success was based in part on the concept of link popularity and PageRank. How many other web sites and web pages link to a given page is taken into consideration with PageRank, on the premise that good or desirable pages are linked to more than others. The PageRank of linking pages and the number of links on these pages contribute to the PageRank of the linked page. This makes it possible for Google to order its results by how many web sites link to each found page. Google's minimalist user interface was very popular with users, and has since spawned a number of imitators.

Researchers at NEC Research Institute claim to have improved upon Google's patented PageRank technology by using web crawlers to find "communities" of websites. Instead of ranking pages, this technology uses an algorithm that follows links on a webpage to find other pages that link back to the first one and so on from page to page. The algorithm "remembers" where it has been and indexes the number of cross-links and relates these into groupings. In this way virtual communities of webpages are found.

[edit]

Challenges faced by search engines

• The web is growing much faster than any present-technology search engine can possibly index (see distributed web crawling).
• Many web pages are updated frequently, which forces the search engine to revisit them periodically.
• The queries one can make are currently limited to searching for key words, which may results in many false positives.
• Dynamically generated sites, which may be slow or difficult to index, or may result in excessive results from a single site.
• Many dynamically generated sites are not indexable by search engines; this phenomenon is known as the invisible web.
• Some search engines do not order the results by relevance, but rather according to how much money the sites have paid them.
• Some sites use tricks to manipulate the search engine to display them as the first result returned for some keywords. This can lead to some search results being polluted, with more relevant links being pushed down in the result list.

[edit]

See also

• Search engine spammer
• History of the Internet
• List of search engines
• Data mining
• Metasearch engine

[edit]

External links

• Articles and information on search engines and how they work (http://www.searchengineforums.com)
• On Search, the Series (http://www.tbray.org/ongoing/When/200x/2003/07/30/OnSearchTOC); Tim Bray; 2003 — A series of essays on search engine techniques.
• Search Engine History (http://blog.outer-court.com/history/) — From Blaise Pascal to Google
• Search engine makes social calls (http://www.nature.com/nsu/020304/020304-8.html) (March 7, 2002; Nature.com)

Retrieved from "http://en.wikipedia.org/wiki/Search_engine"

Categories: Information technology | Internet search engines

Internet research

From Wikipedia, the free encyclopedia.

Internet research is the practice of using the Internet for research. To the extent that the Internet is widely and readily accessible to hundreds of millions of people, in many parts of the world, and can provide practically instant information on most topics, it has a profound impact on the way in which ideas are formed and knowledge is created.

Research is a broad term. Here, it is used loosely to include any activity where a topic is identified, and an effort is made to actively gather information for the purpose of furthering understanding (no matter how trivial-seeming the subject).

Prior to the Internet, and particularly, the World Wide Web, print - books, magazines, newspapers, and other printed publications - was the primary source of in-depth information in the most of the world. In print, the book is the basic research unit. Consulting several books on a topic, and related subjects, was the usual research method for most people.

Compared to the Internet, print physically limits access to information. A book has to be identified, then actually obtained. On the Net, the Web can be searched, and typically hundreds or thousands of documents can be found on a topic, within seconds. In addition, email (including mailing lists), online discussion forums (aka message boards, BBS's), and other personal communication facilities (instant messaging, IRC, newsgroups, etc) provide direct access to experts and other individuals with relevant interests and knowledge.

As the Internet continues to expand, Internet research could become the predominant mode of informing ourselves. More people will form ideas based on what they believe is their active "research" ("looking it up", "reading up about it"), rather than more-or-less passively (environmentally) acquired information (the daily news, "someone told me about", "saw an article on").

However, though books are nowadays produced using a digital version of the content, for most books such a version is not available on internet. Thus, not all valuable information is on internet.

[edit]

Questions & Notes

• Print articles are appearing with a bibiliography entirely referencing online sources (ie: written using only Web resources) - is this "credible"? Is an article trusted more, or less, or the same?
• How can the quality of available information measured? (ie: On a practical level, given a range of everyday subjects, what is the difference between the available info found by visiting a fair-sized library, and on the Web?)
• Trusted sources: how can info be vetted? (eg. Wikipedia)
• What effect does keyword/key phrase searching (combined with instant access) have on the way information is located?
• Facts and figures: What percentage of "essential" texts are available online? What materials are available only online (eg: full text of out-of-print books)? How has the (non-fiction) print publishing business been affected?
• How have people's active research habits been changed (seeking specific info, vs passive "research" from, for example, current media and casual conversation)?
• What effect is the availability of Internet research having on regular people (not students, professional researchers, etc) - are people feeling more empowered? Putting more thought into things? Is the level of expectation for factual presentations (eg: the nightly news) affected?
• It is also important to distinguish between 'looking' for information and 'originating' information. Much of what we think of as 'research' is probably best labelled as 'searching' for information. Genuine Internet research - that is, researching the Internet, is quite different.

[edit]

Methodology

There are a number of research methodologies and theoretical approaches that are recommended for Internet Research including:

• Visual Ethnography
• Content Analysis
• Discourse Analysis
• Statistical Sampling
• Survey Research
• Action Research
• Marxist Approaches
• Habermasian (Public Sphere Approaches)
• Feminist Research

[edit]

Internet Research Ethics

Internet Research Ethics is becoming more important now that Institutional Research Boards (IRBs) are having to approve research in Universities.

Broadly speaking there are the following ethical approaches

1. Consequentialist (Or Utilitarian) Ethics

2. Deontological Ethics

3. Ethics of Care

4. Virtue Ethics

5. Open Source Ethics

[edit]

References

Some useful reference texts are:

Anscombe, G. E. M. (1958). Modern Moral Philosophy. Philosophy, 33.

AoIR. (2001). Ethics Working Committee: Preliminary Report for Ethics, from http://aoir.org/reports/ethics.html

Berry, D. M. (2004). Internet Research: Privacy, Ethics and Alienation - An Open Source Approach. The Journal of Internet Research, 14(4).

Boehlefeld, S. (1996). Doing the Right Thing: Ethical Cyber Research. The Information Society, 12(2)(2).

Ess, C. (2001). Internet Research Ethics, from http://www.nyu.edu/projects/nissenbaum/projects_ethics.html

Jones, S. (1999). Doing Internet Research: Critical Issues and Methods for Examining the Net. London: Sage.

King, S., A. (1996). Researching Internet Communities: Proposed Ethical Guidelines for the Reporting of Results. The Information Society, 12(2)(2).

World Wide Web

From Wikipedia, the free encyclopedia.

Graphic representation of the world wide web around Wikipedia

The World Wide Web ("WWW", or simply "Web") is an information space in which the items of interest, referred to as resources, are identified by global identifiers called Uniform Resource Identifiers (URI).

[edit]

Basic terms

Hypertext is viewed using a program called a web browser which retrieves pieces of information, called "documents" or "web pages", from web servers and displays them, typically on a computer monitor. One can then follow hyperlinks on each page to other documents or even send information back to the server to interact with it. The act of following hyperlinks is often called "surfing" or "browsing" the web. Web pages are often arranged in collections of related material called "web sites."

Although the English word worldwide is normally written as one word (without a space or hyphen), the proper name World Wide Web and abbreviation WWW are now well-established even in formal English. The earliest references to the Web called it the WorldWideWeb (an example of computer programmers' fondness for intercaps) or the World-Wide Web (with a hyphen, this version of the name is the closest to normal English usage).

[edit]

Origins

See also: History of the Internet

The Web can be traced back to a project at CERN in 1989 when Tim Berners-Lee and Robert Cailliau built ENQUIRE (short for Enquire Within Upon Everything, a book Berners-Lee recalled from his youth). While it was rather different from the Web we use today, it contained many of the same core ideas (and even some of the ideas of Berners-Lee's next project, the Semantic Web). Berners-Lee mentions that much of the motivation behind the project was so that he could access library information that was scattered on several different servers at CERN.

Tim Berners-Lee published a more formal proposal for the actual World Wide Web on November 12, 1990 [1] (http://www.w3.org/Proposal) and wrote the first web page [2] (http://www.w3.org/History/19921103-hypertext/hypertext/WWW/Link.html) on November 13 on a NeXT workstation. Over Christmas of that year Berners-Lee built all the tools necessary for a working Web [3] (http://www.w3.org/People/Berners-Lee/WorldWideWeb), the first actual web browser (which was a web-editor as well), and the first web server. On August 6, 1991, he posted a short summary of the WorldWideWeb project (http://groups.google.com/groups?selm=6487%40cernvax.cern.ch) on the alt.hypertext newsgroup.

The primary underlying concept of hypertext came from earlier efforts, such as Ted Nelson's Project Xanadu and Douglas Engelbart's oN-Line System (NLS). Both Nelson and Engelbart were in turn inspired by Vannevar Bush's microfilm-based "memex," which was described in the 1945 essay As We May Think. Berners-Lee's brilliant breakthrough was to marry hypertext to the Internet. In the process he developed a system of globally unique identifiers for resources on the Web and elsewhere: the Uniform Resource Identifier.

The World Wide Web had a number of differences of hypertext systems that were then in place.

• The WWW required only unidirectional links rather than bidirectional ones. This made it possible for someone to link to another resource without action by the owner of that resource.

• Unlike certain applications such as HyperCard or Gopher, the World Wide Web was non-proprietary, making it possible to develop servers and clients independently and to add extensions without licensing restrictions.

On April 30, 1993 CERN announced (http://info.web.cern.ch/info/Announcements/CERN/2003/04-30TenYearsWWW/Declaration/Page1.html) that the World Wide Web would be free to anyone, with no fees due.

[edit]

The three standards

The Web is made up of three standards: The Uniform Resource Locator (URL), which specifies how each page of information is given a unique "address" at which it can be found; Hyper Text Transfer Protocol (HTTP), which specifies how the browser and server send the information to each other, and Hyper Text Markup Language (HTML), a method of encoding the information so it can be displayed on a variety of devices. Berners-Lee now heads the World Wide Web Consortium, which develops and maintains these standards and others that enable computers on the Web to effectively store and communicate all kinds of information.

[edit]

Beyond text

Screenshot of the very first WWW browser running on a NeXT system

The initial "www" program at CERN displayed styled text and images, and it was a WYSIWYG HTML editor as well as the browser.

As it ran only on NeXT machines, CERN released a simple, text-only version to the world. Some journalists first encountered the Web through the text browser written by Nicola Pellow and this engendered a myth that the Web was text-only until Mosaic came along. The Web had graphics from the start, at least for NeXT users. However, Mosaic did introduce the ability to combine text and images in a single page.

Meanwhile, browsers such as Tony Johnson's "Midas" and Pei-Yuan Wei's Viola (1991) added the ability to display graphics also on other Unix machines. Marc Andreessen of NCSA released a browser called "Mosaic for X" in 1993 that sparked a tremendous rise in the popularity of the Web among novice users. Andreesen went on to found Mosaic Communications Corporation (now Netscape Communications, a unit of Time Warner). Additional features such as dynamic content, music and animation can be found in modern browsers.

Frequently, the technical capability of browsers and servers advances much faster than the standards bodies can keep up with, so it is not uncommon for these newer features to not work properly on all computers, and the web as seen by Netscape or Mozilla Firefox is not quite the same as the web seen by Internet Explorer. The ever-improving technical capability of the WWW has enabled the development of real-time web-based services such as webcasts, web radio and live web cams.

[edit]

Java and JavaScript

Another significant advance in the technology was Sun Microsystems' Java programming language, which enabled web servers to embed small programs (called applets) directly into the information being served that would run on the user's computer, allowing faster and richer user interaction.

JavaScript however, is a scripting language that was developed for Web pages. While its name is similar to Java it was developed by Netscape and not Sun Microsystems. In conjunction with the Document Object Model, JavaScript has become a much more powerful language than its creators originally envisioned. Sometimes its usage is expressed under the term DHTML, or Dynamic HTML, to emphasize a shift away from static HTML pages.

[edit]

Sociological implications

The exponential growth of the Internet was primarily attributed to the emergence of the web browser Mosaic, followed by its commercial offspring Netscape Navigator, during the mid-1990s.

It brought unprecedented attention to the Internet from media, industries, policy makers, and the general public.

Eventually, it led to several visions of how our society might change, although some point out that those visions are not unique to the Internet, but repeated with many new technologies (especially information and communications technologies) of various era.

Because the web is global in scale, some suggested that it will nurture mutual understanding on a global scale.

[edit]

Publishing web pages

The web is available to individuals outside mass media. In order to "publish" a web page, one does not have to go through a publisher or other media institution, and potential readers could be found in all corners of the globe.

Unlike books and documents, hypertext does not have a linear order from beginning to end. It is not broken down into the hierarchy of chapters, sections, subsections, etc.

Many different kinds of information are now available on the web, and for those who wish to know other societies, their cultures and peoples, it became easier. When travelling in a foreign country or a remote town, one might be able to find some information about the place on the web, especially if the place is in one of developed countries. Local newspapers, government publications, and other materials are easier to access, and therefore the variety of information obtainable with the same effort may be said to have increased, for the users of the Internet.

Although some websites are available in multiple languages, many are in the local language only. Also, not all software supports all special characters, and RTL languages. These factors would challenge the notion that the World Wide Web will bring a unity to the world.

The increased opportunity to publish materials is certainly observable in the countless personal pages, as well as pages by families, small shops, etc., facilitated by the emergence of free web hosting services.

[edit]

Statistics

According to a 2002 survey of 2,024 web pages [4] (http://www.netz-tipp.de/languages.html), by far the most Web content was in English: 56.4%; next were pages in German (7.7%), French (5.6%) and Japanese (4.9%).

According to a 2001 study [5] (http://www.brightplanet.com/technology/deepweb.asp), there were more than 550 billion documents on the web, mostly in the "invisible web".

[edit]

Speed issues

Frustration over congestion issues in the Internet infrastructure and the high latency that results in slow browsing has lead to an alternative name for the Web: the World Wide Wait. Speeding up the Internet is a still undergoing discussion over the use of peering and QoS technologies. Other solutions to reduce the World Wide Wait can be found on W3C (http://www.w3.org/Protocols/NL-PerfNote.html).

[edit]

Academic conferences

The major academic event covering the WWW is the World Wide Web series of conferences, promoted by IW3C2 (http://www.iw3c2.org). There is a list (http://www.iw3c2.org/Conferences/Welcome.html) with links to all conferences in the series.

[edit]

Pronunciation of "www"

Most English-speaking people pronounce the 9-syllable letter sequence www used in some domain names for websites as "double U, double U, double U", but many shorter pronunciations can be heard: "triple double U", "double U, double U" (omitting one W), "dub, dub, dub", "hex u", etc. Perhaps a shorter pronunciation will become standard usage in the future.

[edit]

See also

• History of the Internet
• Semantic Web
• Media studies
• Smartphone
• List of websites
• Search Engine
• Hypertext
• Les Horribles Cernettes (first picture ever to be clicked on in a web browser)

[edit]

External links

• Architecture of the World Wide Web, Volume One (http://www.w3.org/TR/2004/REC-webarch-20041215/)
• Complete Guide to Internet Statistics and Research, including WWW (http://internet-statistics-guide.netfirms.com)
• Internet Statistics: Growth and Usage of the Web and the Internet (http://www.mit.edu/people/mkgray/net/)
• 31 December, 2003, BBCNews: Web's inventor gets a knighthood (http://news.bbc.co.uk/1/hi/technology/3357073.stm) The inventor of the world wide web, Tim Berners-Lee, has been awarded a knighthood for his pioneering work. Citat: "...The original idea of the web was that it should be a collaborative space where you can communicate through sharing information..."
• http://dmoz.org/Computers/Internet/Web_Design_and_Development/
• First known web page (http://www.w3.org/History/19921103-hypertext/hypertext/WWW/TheProject.html)
• Tutorials for Beginning Computer Users: How the Web Works (http://www.bleepingcomputer.com/forums/tutorial45.html)
• Red vs. Blue's Reality vs. the Internet (video) (http://demandmedia.net/story/2004/8/13/18037/9310)

Retrieved from "http://en.wikipedia.org/wiki/World_Wide_Web"

Categories: Information technology | Internet | Digital media | World Wide Web