Saturday 24 September 2011

Kernel: Monolithic vs Micro


Kernel: Monolithic vs Micro

Kernel is the core of the Operating System(OS), and roughly consists of two parts: the kernel space (privileged mode) and the user space (unprivileged mode)without that, protection between the processes would be impossible.

There are two different concepts of kernel: Monolithic kernel and Micro-kernel. The older approach is the Monolithic kernel, of which Unix, MS-DOS and Mac OS are typical represents of. It runs basic system services like process, memory management, interrupt handling, I/O communication, file systems in kernel space.

Monolithic kernel

The inclusions of all basic services in kernel space has three disadvantages:

Ø       The kernel size
Ø       Lack of extensibility
Ø       Bad maintainability

Bug fixing or addition of new featured means re-compilation of the whole kernel which consumes lot of time as well as lot of memory.

To overcome the above disadvantages micro-kernels was introduced in late 1980’s. The concept was to reduce kernel to basic process and I/O, memory system services reside in the user space (they are called as servers). There are servers for managing issues, another server for managing drivers and so on... Communications happens through Context switching where user processes are allowed to kernel space and then exit. Messaging system was introduced which has independent communication for process rather than context switching which consumes more memory.

Micro-kernel


First generation of micro-kernel was had a lot of drawback concerning IPC, device drivers, communication stacks which resulted in larger kernel size which was slower in execution.

Later was thought to create a pure micro-kernel where it fits to processor’s first level cache. Second generation micro-kernel (L4) was highly optimized which results in a very good I/O performance.


Important Parameters in Comparisons:

Memory management:

Monolithic kernel:
Monolithic kernels implement everything needed for memory management in kernel space. This includes allocation strategies, virtual memory management, and page replacement algorithms.

Monolithic - Memory Management

Micro-kernel:
Micro-kernel L4 has got three memory management primitives: map, grant and flush. A process maps memory pages to another process if he wants to share these pages. When a process grants pages to another process, he cannot access them anymore. Flushing regains granted and mapped memory pages.

Micro-kernel - Memory Management


The micro-kernel reserves the whole system memory at startup to one process, the base system process, which resides (like all other processes) in user space. If a process needs memory, he doesn't have to take the way through the kernel anymore, but directly asks the base system process. Because every process can only grant/map/flush the memory pages it owned before, memory protection still exists.

I/O Communication:

Monolithic kernel:
I/O communication works through interrupts, issued by or sent to the hardware. Monolithic kernels run device drivers inside the kernel space. Hardware interrupts are directly handled by kernel processes. To add or change features provided by the hardware, all layers above the changed layer in the monolithic kernel also have to be changed in the worst case.

The concept of so called modules was introduced to achieve more independence and separation from the kernel. One module represents (parts of) a driver and is (un)loadable during run time. That way, drivers which are not needed by the system are not loaded and memory is preserved. But kernel modules are still binary kernel-dependent. If concepts change too much inside the monolithic kernel, modules need not just a recompilation, but a complete code adoption.

Micro-kernel:
The micro-kernel approach doesn't handle I/O communication directly, It only ensures the communication. Requests from or to the hardware are redirected as messages by the micro-kernel to the servers in user space. If the hardware triggers an interrupt, the micro kernel sends a message to the device driver server, and has nothing more to do about it. The device driver server takes the message and sends it to the right device driver. That way it is possible to add new drivers, exchange the driver manager without exchanging drivers or even exchange the whole driver management system without changing any other part of the system.

Tuesday 20 September 2011

Linux: A bit of history

MINIX(1984), created from scratch by Andrew Tanenbaun, for educational purposes in order to reach how to design and implement  OS.

Andrew Tanenbaun

MINIX was mainly designed for educational purpose rather than professional activities. MINIX was mainly running on most successful 8086 platform. The advantage of this kernel was the source code was available to anyone from Tanenbaun's teaching books on OS.

1990, FSF(Free Software Foundations) and GNU project, motivated many programmers to promote quality and free distributed software's.Aside from software, work was being done on the kernel OS known as HURD.

1991, Finnish student Linux Torvalds presented version 0.0.1 of his kernel which he called as "Linux" designed for i386 architectures and offered GPL license to community of programmers, Internet community for testing,they liked it and stared helping with its development.

Linux Torvalds

Distinguish Linux from other OS(UNIX) :


1. Open Source: Anyone can have access to source code, change them and create new version that can be shared under GPL license.


2. Portability: Independent of any architecture but with a 'C' compiler such as GNU gcc.GNU/Linux runs almost all architectures like Intex X86, IA64, AMD x86, x86_64, Sun's SPARC, Power PC, IBM S390, ARM ....


3. Monolith type kernel: Design of the kernel is joined into a single piece but is conceptually modular in its different tasks. Problem with monolith was when they grow they become very large and untreatable for development; DDL were used to try to resolve this.


4. DLL: These make it possible to have the parts of OS, such as filesystems, devices, as external parts that are loaded(or linked) with the kernel at run-time on demand.
This would simplify the kernel as these functionalities as elements that can be separately programmed.


5. Projects succeeded with Linux kernel:
    - people of FSF, with the GNU utility software and above all with GCC C compiler joined projects like XFree, Gnome, Kde.
    - Internet development with projects like Apache web server, Mozilla navigator, MySQL, and PostgreSQL databases ended up giving Linux kernel sufficient coverage to compete with proprietary systems.

New companies that created GNU/Linux distributions(packaging of kernel + applications) and supported it such as Red Hat, Mandrake, Suse, made an unstoppable growth that we are witnessing today.

Highlights of GNU's contribution:

    - C and C++ compilers
    - bash shell
    - Emacs editor
    - Postscript interpreter
    - Standard C library (glibc)
    - Debugger (gdb)
    - Makefile (GNU make)
    - Assembler
    - Linker

In further article we see how above scientists made an approach and what would be the functionality of Monolithic and Micro-kernel
 

Friday 16 September 2011

UNIX: A bit of history

UNIX started back in 1969 in BTL(Bell Telephone Labs) and AT&T.These were just withdrawn from a project called MULTICS,which was designed to create an OS to support thousands of users simultaneously.BTL,General Electric,MIT were involved in this project,as it failed MIT was withdrawn from the project


Two engineers who were involved in MULTICS were Ken Thompson and Dennis Ritchie, found a computer which had an assembler and loading program.They developed tests on these assembler with the kernel.
8684d799768f77fb2278c7e01df7003e_1M.png
Ken Thompson(Left) and Dennis Ritchie


1969, Thompson had an idea of writing a file system for the created kernel so that files are stored in an ordered form in a hierarchical way. As progress made on the system design few more BTL engineers joined the project. The system became too small to work on and to purchase PDP machine with an agreement of creating a new text processor.


when new machine arrived they gave only CPU and memory not disk or OS.Thompson unable to wait designed a RAM disk in memory and used half of memory as disk other half for OS that he was designing. Once the disk arrived they stared working on OS and promised text processor(Troff)which was used to create UNIX "man pages". BTL started using UNIX with new text processor.


Another important characteristic was that UNIX was independent of hardware architecture. 1971, external users wanted to document what was being done resulted in UNIX programmers manual signed by Thompson and Dennis. UNIX installations continued to grow to about 50


End 1973,decision to present results at a conference on OS, so various IT  centers, universities asked for copies of UNIX. AT&T did not support, so users had to unite and share their knowledge by forming community(USENIX). AT&T decided to cede UNIX to universities,but no support.


University to obtain license was Berkley where Thompson studied.1975 Thompson returned to Berkley as a teacher bringing with him the latest kernel.Two newly graduated students, Chuck Haley and Bill Joy joined him and started to work together in UNIX implementation.





BillJoy.jpg
Bill Joy



The major disappoinment were the editors, Joy perfected an editor called "EX" later the same into "vi".Two developed Pascal language compiler which was added into UNIX.Demand continued to grow for UNIX which made Joy to produce "BSD UNIX"


1978, BSD had a license regarding its price for distribution so that new users could end making some changes or incorporating features,selling copies after certain period of time.


Joy made further changes to "vi" in such a way that editor was independent of terminal. He created the TERMCAP system programs could get executed irrespective of the terminals using interface. 1977, UNIX was running on PDP machines, that year adaptations were made for machines of the time such as Interdata and IBM.More versions offered as it included:awk,lint,make,uucp,C compiler designed by Kernighan and Dennis Ritchie which had created to re-write most UNIX that was initially running in assembler.Also included with bash,find,cpio,expr


UNIX started to appear from companies such as Xenix,Berkely. AT&T realised that UNIX was only a commercial product and its license prohibited its study in academic institutions in order to protect its commercial secret. until then UNIX source code was used in universities to teach OS


Every one found their own solution for solving problems.Andrew Tanenbaum decided to write a new UNIX-compactable OS without using a sinlge line of AT&T code.He called this OS as MINIX.


Bill Joy decided to leave to a new company called SUN Microsystems where he modified the BSD 4.2 to newly created SUN's UNIX (SunOS). Every company stared using their own UNIX.


- IBM     - AIX
- DEC     - Ultrix
- HP      - HPUX
- SGI     - IRIX
- Apple   - MacOS X


AT&T released final version called UNIX System V(SV),as well as BSD 4.x.Current UNIX versions are either running on SV or BSD.some manufactures specify their UNIX in BSD or SV style.Later which UNIX standards were drawn up hence we find IEEE POSIX, UNIX 97, FHS .. etc



Wednesday 31 August 2011

Linux Based Super Computers

K-Computer:
K-computer is currently the world's fastest supercomputer. It is developed by Fujitsu at the RIKEN Advanced Institute for Computational Science campus in Kobe, Japan. K-computer gives a peak performance of a mind-blowing 8.16 petaflops toppling Tianhe-1A off its number one spot. This uses 68,544 2.0 GHZ 8-core SPARC 64 VIIIfx processors packed in 672 cabinets, for a total of 548,352 cores. In layman's term, K-computer's performance is almost equivalent to the performance of 1 million desktop computers. The file system used here is an optimized parallel file system based on Lustre, called Fujitsu Exabyte File System. Being such a high-performer, this supercomputer consumes about 9.8 MW of power, that's the amount of power that would be enough to light 10,000 houses


Tianhe-IA:
 
Tianhe-IA that was developed by the Chinese National University of Defense in Changsha, Hunan. Tianhe-1 stands for "Milky Way number 1" in Chinese. Both the supercomputers use Linux as their operating system and are undoubtedly the meanest machines China has ever produced he 88 million dollar beast consists of 112 computer cabinets, 12 storage cabinets, 6 communication cabinets and 8 I/O cabinets. Each cabinet has 4 frames, each frame having eight blades and a 16-port switching board. The system has 3584 such blades containing 7168 GPUs and 14,336 CPUs Tianhe-1A also uses Lustre file system for its 2 Petabytes storage system. This Chinese marvel has given a peak performance of about 2.5 petaflops and is used in carrying out computations for petroleum exploration and aircraft design

    
Jaguar Cray:

Running on Cray Linux Environment, Jaguar is currently the world's third fastest supercomputer. It has achieved a peak performance of about 1.75 petaflops and was once the world's fastest supercomputer before being overtaken by the Chinese Tianhe-1A in 2010-11. The current model, that is Cray CTX5, is an upgraded version of the popular Cray CTX4. Jaguar has around 224, 256 x86-based AMD Opteron processor cores with 16 GB of memory for each node. The file system used here is an external Lustre file system, which is basically a massively parallel-distributed file system that is used for cluster computing. The world Lustre is derived from a combination of the terms Linux and cluster. The file system is capable of storing over 10 Petabytes of data and has a read/write benchmark of 240 GB/s.


Nebulae:

Nebulae is a research supercomputer located in Shenzhen, Guangdong, China. It runs on Linux and has a theoretical peak performance of around 2.9 petaflops


 



TSUBAME 2.0:

TSUBAME 2.0 is the successor of TSUBAME 1.0, which previously was the fastest supercomputer in Japan. TSUBAME stands for Tokyo Tech Supercomputer Ubiquitously Accessible Mass storage Environment. Tsubame is also the word for a swallow in Japanese that forms an integral part of their logo. To create TSUBAME 2.0, Tokyo Tech partnered with big corporations like HP, Nvidia, Intel, NEC and others. The Japanese marvel has a theoretical peak performance of a whopping 2.4 petaflops making it the 5th fastest supercomputer in the world. It has an aggregated memory bandwidth of 720 Terabytes per second. This is a huge improvement over TSUBAME 1.0 that offered around 17 TB/s. The storage capacity is of 11 Petabytes, which can be expanded. And yes, it runs on Linux


Cielo Cray XE:
This mean machine that was unveiled in May 2010, is the sixth fastest supercomputer in the world. It runs on Linux and is powered by AMD x86-64 Opteron 8 core processor. Cielo is located in Los Alamos National Laboratory in New Mexico, USA and is mainly used for research purposes



Pleiades SGI Altix:

Pleiades is a supercomputer used by NASA to conduct modeling and simulation for their missions. Its performance averages around 1.09 petaflops with a peak of 1.315 petaflops. Loaded with a memory of 185 TB and 111,104 cores, Pleiades is the world's 7th fastest supercomputer. The beast runs on SUSE Linux and has about 6.9 PB of storage space with 12 Direct Data Network (DDN) RAIDs


 
Cray XE6:
 
Housed in DOE's National Energy Research Scientific Computing Center (NERSC), California, Cray XE6 is currently the world's 8th fastest supercomputer. It has achieved a peak performance of 1.5 petaflops and runs on Cray Linux Environment version 3. Specs include, 1536 cores per cabinet with 8 or 12-core 64-bit AMD Opteron 6100 Series processors. XE6 also comes with a Hardware Supervisory System (HSS) that integrates hardware and software components to provide system monitoring, fault identification and recovery

Tera 10:

Built by the French company Bull SA, Tera 100 is Europe's fastest supercomputer. It runs on Red Hat Enterprise Linux and gives an average of 1 petaflops, peaking at 1.25 petaFlops. It is one of the most efficient supercomputers in the world running at an efficiency of 83.7 %. Going back to the specs, Tera 100 comes with 20 Petabytes of storage, 300 TB of memory and the processing power of 140,000 Intel Xeon processor cores. This supercomputer includes specially designed water-cooled doors, which cut electrical consumption to half when compared with traditional air-cooled ones.


 
IBM Roadrunne:

The world's tenth fastest supercomputer, IBM Roadrunner was built by IBM at the Los Almos National Laboratory in New Mexico, USA. It costs around 125 million USD and is the fourth most energy efficient supercomputer in the world. A computer's performance is generally measured in FLOPS, which stands for floating point operations per second. IBM's Roadrunner has a speed of about 1 petaflops(1015) with a top speed of 1.456 petaflops which it reached in November 2008. It uses Red Hat Enterprise Linux along with Fedora as its operating system and occupies almost 6000 sq. ft. of real estate. Roadrunner's main use is to predict whether USA's aging arsenal of nuclear weapons is safe and reliable. It is also used in other fields like financial, aerospace and automotive industries.

The unique thing about Roadrunner is its use of two different processing architectures at the same time, more commonly known as hybrid design. This consists of AMD's Opteron along with IBM's own Powercell 8i. In case your dual core computer's speed was never good enough for you, the IBM Roadrunner boasts of a whopping 122,400 cores.
 


Tuesday 5 July 2011

UNIX to LINUX Migration

This article which I am writing would let you know the things that should be considered before migrating UNIX to Linux

1. Business Objective:
    The motivation among the customers to choose Linux is cost, whether it is driving cost reduction through consolidation of servers or applications or the use of visualization and adaptive infrastructure technologies to increase average server utilization. In all the above cases migration is required only if there is a valid business to do so.
    A new line of business requires new application, escalating hardware maintenance costs, escalating software license or support costs.Hence current cost benefits should be equally factoring in the future opportunities that each option affords.

2. Scope and Availability:   
    The goal might be transition to the Linux,adopting to a virtual environment, upgrading to a later version of the current application, introducing new platform which would be simple according to the project needs.
    Any project should be scoped and planned to an accepted level of risk for the organization.

3. Migration In-house or Outsource:   
    Its all your preference to utilize in-house skills to manage migration or to outsource the project, support for Linux means there are many partnerships available. Hardware vendors wants customers to run Linux and thus provide whitepapers,toolkits,checklists which could be freely downloaded and useful for self assisting analysis
IBM and HP are the two vendors for RHEL Linux.There are Red Hat consultants who can integrate in designing,deploying the best practices.

4. Hardware Options:   
    Depending on the scope of the project, it may be appropriate to retain existing servers, to use redeployed servers, or to provision new servers. X86_64 platforms are providing stunning performance and make up a large portion of Linux platforms.

5. Planning your application:
    you could retain the applications from UNIX to Linux, thus providing the common end-user interface experience.RHEL offers a large application portfolio, which could be later upgraded to take advantage of the functional enhancements.Application could be Database,custom,third-party.

Saturday 4 June 2011

Government Users of Linux

Government Users of Linux

Some of the government agencies that use Linux at all levels(national, state, federal and international)have opted to deploy Linux across their computer for the below reasons:
1. Preferring the open-source benefits of the OS
2. Others are financial, as Linux is typically far less expensive than buying a license for Windows

Here are some of the governing bodies that now run Linux on their computers

1.U.S. Department of Defense: "single biggest install base for Red Hat Linux" in the world. http://bit.ly/1VEOAN

2. U.S. Navy Submarine Fleet: the US Navy nuclear submarine fleet is using GNU/Linux http://bit.ly/bNk9uJ

3.The City of Munich, Germany: Uses Debian which was customized to meet Munich's municipal computing needs http://bit.ly/moPt5K

4.Spain: its own customized Linux distribution called LinEx based on Debian http://bit.ly/mUE9Px

5.Federal Aviation Administration: migrating to Red Hat Enterprise Linux saved 15 million dollars. http://bit.ly/mxL4XJ

6.French Parliament:Started using Ubuntu Linux from November 2006. http://zd.net/jh5FOi

7.Industrial and Commercial Bank of China:Linux became "the basis for its web server and a new terminal platform" http://bit.ly/b9EVfV

8.Pakistani Schools & Colleges: http://bit.ly/kbelN4

9.Cuba: Has its own Linux called "Nova"

10.Macedonia's Ministry of Education and Science: Uses Ubuntu workstation http://bit.ly/11r1fi

11.U.S. Postal Service: While the Postal Service ran Windows NT on its servers until the bitter end, they then switched to using over 900 Linux clusters spread throughout the country for use in sorting the nation's bulk mail

12.U.S. Federal Courts: U.S. Federal Courts rely on Linux for all manner of administrative tasks, including "case management, case tracking, finance and accounting, probation and pretrial services"

13.Government of Mexico City: government of Mexico City had concluded that "they can no longer justify the ever rising cost of Microsoft Windows when the cost of Linux software is very low"

14.Garden Grove, California: it saved so much money that they later decided to roll out Linux across the city, including on some desktop systems

16.Largo, Florida: "Largo Loves Linux More Than Ever" explains how the Floridian city came to rely so heavily on Linux software. After having such great success running city computers on Linux, Largo's municipal government soon thereafter was "talking about Linux-based terminals in all the city's police cars.

17.Czech Post: Perhaps taking a cue from the U.S. Postal Service, the Czech Republic's own post office successfully migrated to Linux in 2005. The chosen distribution of Linux (SuSe) now runs on "4,000 servers

Educational Users of Linux

18.Russian Schools: In 2007, the nation of Russia announced that all its schools would begin running Linux software

19.German Universities:
German students plus thousands of staff at 33 German universities will now be supported by Linux systems from Novell." SuSe Linux Enterprise Desktop was the specific distribution chosen, evidently for the "more flexible IT architecture" that it provides "when compared to other proprietary software."

20.The Phillipines: after a successful deployment of 13,000 Fedora Linux systems from a government grant, plans are underway to roll out another 10,000 based on Ubuntu

21.Georgia: began running all its school computers and LTSP thin clients on Linux, mainly using Kubuntu, Ubuntu and stripped Fedora-based distros

22.The Indian State of Tamil Nadu: the Indian state of Tamil Nadu decided instead to "distribute 100,000 Linux laptops to students there."

23.Switzerland Schools: Wikipedia also reports that Switzerland converted 9,000 of its computers to using Linux and OpenOffice.org's suite of office productivity tools

24.Bolzano, Italy: The town of Balzano in Italy (with a student population of 16,000) reportedly switched to using a customized distribution of Linux.

25.Kerala, India: government and government-aided high schools in the state will no longer use the Windows platform for computer education. Instead, they have switched over to the free GNU/Linux software

26.Indiana Schools: more than 20,000 Indiana students are now Linux-enabled under a state grant program to roll out low-cost, easy-to-manage workstations

Business Users of Linux

27. Novell:Longtime software and services company Novell announced in 2006 that it was undergoing a company-wide migration from Windows to Linux on employee desktop computers

28. Google:Believe it or not, the gigantic, ever-growing cluster of servers that power Google's search and other apps runs Linux

28. IBM: IBM Supports Linux 100%. In the last decade, perhaps no larger company than IBM has contributed more to the success of Linux, both financially and developmentally

29. Panasonic: Electronics giant Panasonic is another household name company to use Linux in powering some of its operations

30.Virgin America:Virgin America, a low-cost U.S. airline run by entrepreneurial big-shot Richard Branson, uses Linux to power its in-flight entertainment The entertainment system (called RED) is powered by Red Hat and Fedora specifically, and was reportedly chosen because it is "very stable and agile.

31. Cisco: Cisco Systems, the computer networking and routing giant, switched to Linux after vowing to use Microsoft's Active Directory solution for its servers In an imfamous turn of events, however, Cisco's own IT staff could not get its network printign to work properly using Windows NT and were thus forced to switch to Linux

32.Conoco Phillips: Never let it be said that Linux is a fringe operating system for inconsequential gizmos and gadgets. No stronger proof to the contrary exists that ConocoPhillips, which proudly uses Linux to power a massive (and massively important) cluster of servers aimed at exploring the earth for new sources of untapped oil

33.Omaha Steaks: Omaha Steaks, a popular catalog-oriented steak retailer, switched to open-source Linux in 2001

34. Amazon: Online book and electronics retail behemoth Amazon.com is said to "use Linux in nearly every corner of its business"

35.Peugeot: European car maker Peugeot announced in 2007 that it was set to deploy up to 20,000 copies of Novell Desktop Linux and 2,500 copies of Suse Linux Enterprise Server

36.Wikipedia: Popular online encyclopedia Wikipedia is another staunch supporter of Linux, having switched to Ubuntu in 2008 after a lengthy tenure using Red Hat and Fedora prior to that

37.New York Stock Exchange:
The New York Stock Exchange is another perhaps unexpected business user of Linux.
London's stock exchange was also "abandoning the failed Windows platform", it was stated that New York's exchange already used Linux to power its trading platform and furthermore that it "seems to be doing quite nicely. that it was Red Hat Enterprise Linux, specifically, that the NYSE ran on its trading platform.

38 Burlington Coat Factory: Burlington Coat Factory, a retailer with 280 individual stores across 42 states, run Linux in their distribution centers and "a few new stores"

39.Raymour and Flannigan: Raymour and Flannigan's transition to Linux as "a major transformation" for the Syracuse-based furniture retailer, who switched all its servers to Linux back in 2002

40.Tommy Hilfiger: Tommy Hilfiger "chose eOneGroup and Linux for its new e-business infrastructure" way back in 2001. Company representatives were quoted as saying that "we saved significantly on the time and expense of deploying this total infrastructure

41.Toyota Motor Sales: "30 dealer pilot roll-out" of a system using Linux to connect car dealerships to Toyota's factories. The system was a "web based system from the ground up, and will be handling 30 different functions including parts ordering, warranties, sales transactions and repairs."

42.Travelocity: Travelocity (funny gnome guy and all) is yet another Internet business powered by Linux servers. "to improve our flexibility and really decrease our time to market" as the chief reasons for choosing Linux over other alternatives

43. Netbooks:While Net-books are still frequently sold with Microsoft Windows installed, they are shipped with Linux more than perhaps any other mass-market laptop around

44.Dell Model: In recent years (particularly 2007-2008) distributions of Linux like Ubuntu have placed a higher than ever priority on user friendliness in efforts to capture some of the Windows market

45.CERN: CERN uses Scientific Linux on a massive scale for mission-critical applications