|
Grid Computing for Developers
Many industry analysts believe that grid computing will be the next big technology wave. It is so promising that major technology companies such as IBM, Microsoft, Oracle, and Sun are allocating significant resources for grid research and development. Grid Computing for Developers explores the evolution of grid computing from the early distributed
computing paradigm to the modern grid environments. Most of today's grid environments will be examined including OGSA, Resource Managers, and MPI. The book also explores today's de-facto standards for grid computing including a comprehensive guide to the Globus Toolkit for developers covering installation, system administration, development guides for grid services, open grid services architectures for OGSI and OGSA-DAI, commodity grid kits for Java, Perl, and OGSA-C, and performance evaluation tests. A significant amount of source code samples accompany each chapter. Grid Computing for Developers is an ideal resource for computer scientists and researchers, graduate students, and IT professionals looking to further expand their grid computing skills.
CONTENTS:
Dedication v
Preface xxi
Part I Theory and Foundations 1
1 The Roadmap to High-Performance Computing 3
1.1 Evolution of Grid Technologies 3
1.2 The Grid in a Nutshell 5
1.2.1 What Can Be Called a Grid Application 6
1.3 Distributed Computing Models 6
1.3.1 Internet Computing 6
1.3.2 Peer-to-Peer (P2P) 7
1.3.3 Grid Architectures 8
1.4 Computing on Demand (COD) 11
1.4.1 How It Works 11
1.4.2 User Authorization 12
1.4.3 Limitation 12
1.5 Grids in Science and Technology 12
1.5.1 Grid Consortiums and Open Forums 12
1.5.2 Grids in Science and Engineering 13
1.6 Summary 15
2 Enterprise Computing 17
2.1 Existing Enterprise Infrastructures 18
Contents
vii
2.2 Integration with Grid Architectures 19
2.2.1 Workload and Policy Management 20
2.2.2 Shared Services 21
2.2.3 Stateless Business Components 21
2.2.4 Client State Management 21
2.2.5 Pooled Resources and Lightweight Server Processes 22
2.2.6 Database Concurrency Maximization 22
2.3 Grid-Enabled Resource Management Models 23
2.3.1 Hierarchical 24
2.3.2 Abstract Owner Model (AO) 24
2.3.3 Economy/Market Model 25
2.4 Enter Open Grid Services 25
2.4.1 Grids on the Enterprise 26
2.4.2 Challenges Facing Grid Computing on the Enterprise 28
2.4.3 The Jump to Web Services Resource Framework (WSRF) 29
2.5 The Competitive IT Organization 29
2.6 Computational Economy 30
2.6.1 Why Computational Economy? 30
2.6.2 Economy-Driven Grid Resource Management 32
2.6.3 Resource Management Models 33
2.6.4 Grid Architecture for Computational Economy (GRACE) 34
2.7 Summary 35
3 Core Grid Middleware 39
3.1 P2P 41
3.1.1 Definition 41
3.1.2 Advantages and Disadvantages 41
3.1.3 P2P vs. the Client-Server Model 42
3.1.4 P2P Architectures 43
3.2 Globus 45
3.3 Grid Computing and Business Technologies (GRIDBUS) 46
viii Contents
3.4 Summary 46
Part II Grid Middleware 49
4 Grid Portal Development 51
4.1 Resource Management 52
4.1.1 Apache Jetspeed 53
4.1.2 A Resource Management JSP Portlet for Jetspeed 53
4.1.3 WebSphere Portal Server v5.0 (WPS) 89
4.1.4 IBM Portlet API 89
4.1.5 Job Submission Portlet for WPS 91
4.1.6 Portlet Program 91
4.1.7 Portlet View JSP 95
4.1.8 Testing 98
4.2 Data Management Portlets 99
4.2.1 Global Access to Secondary Storage (GASS) 99
4.2.2 GridFTP 100
4.2.3 Remote I/O 100
4.2.4 Globus Executable Management (GEM) 100
4.2.5 Remote Data Access Portlet for Jetspeed 100
4.2.6 Remote Data Access Portlet for IBM WPS 107
4.3 Troubleshooting 115
4.3.1 Certificate Not Yet Valid Errors 115
4.3.2 Defective Credential Errors 115
4.4 Summary 116
5 Schedulers 119
5.1 An Overview of Popular Schedulers and Grid Middleware Integration 120
5.1.1 Job Schedulers versus Job Managers 121
5.2 Portable Batch System - OpenPBS 121
5.2.1 Overview 121
5.2.2 OpenPBS in Red Hat Linux (RHL) 9 122
Contents ix
5.2.3 Configuring PBS on a Single Machine Cluster 123
5.2.4 Tweaking PBS GUIs for RHL 9 124
5.2.5 Configuring the Globus Toolkit 3.2 PBS Job Scheduler 125
5.2.6 Customizing the PBS Job Manager 126
5.2.7 Troubleshooting PBS MMJFS Runtime Errors 127
5.2.8 Customizing PBS Scheduler Execution Parameters with RSL 131
5.3 Silver/Maui Metascheduler 134
5.3.1 Reservations 135
5.3.2 Fairshare 135
5.3.3 Backfill 135
5.3.4 Job Prioritization 136
5.3.5 Throttling Policies 136
5.3.6 Quality of Service (QoS) 136
5.3.7 Node Allocation 136
5.4 Sun Grid Engine (SGE) - N1 Grid Engine 6 137
5.4.1 Workload Management 137
5.4.2 Architecture 137
5.4.3 Step-by-Step SGE Installation Transcript 138
5.4.4 Installation Troubleshooting Tips 140
5.4.5 Installation Verification 142
5.5 Condor-G 143
5.5.1 Condor Features 143
5.5.2 Condor-MMJFS Installation Transcript 144
5.6 MMJFS Integration with Other Job Schedulers 147
5.6.1 Step 1: Create a Job Manager Script 148
5.6.2 Step 2: Update the GT 3.2 Deployment Descriptors 161
5.6.3 Step 3: Verification 165
5.6.4 Integration Troubleshooting 170
5.6.5 Packaging Advice 176
5.7 Factors for Choosing the Right Scheduler 178
5.7.1 Features 178
x Contents
5.7.2 Installation, Configuration, and Usability 178
5.7.3 User Interfaces 178
5.7.4 Support for Open Standards 178
5.7.5 Interoperability 179
5.7.6 Organization's Requirements 179
5.7.7 Support for Grid Middleware 179
5.8 Summary 180
6 Open Grid Services Architecture (OGSA) 181
6.1 Web Services versus Grid Services 182
6.2 OGSA Service Model 184
6.3 Standard Interfaces 184
6.4 Other Service Models 185
6.5 Service Factories 186
6.6 Lifetime Management 187
6.7 Handles and References 187
6.8 Service Information and Discovery 187
6.9 Notifications 188
6.10 Network Protocol Bindings 188
6.11 Higher-Level services 189
6.12 Other Interfaces 189
6.13 Hosting Environments 189
6.14 Virtual Organizations (VOs) 190
6.15 The Globus Toolkit 191
6.15.1 Core Services 192
6.15.2 Security 192
6.15.3 Data Management 192
6.16 Sample Grid Services: Large Integer Factorization 193
6.16.1 Overview: Large Integer Factorization and Public
Key Cryptosystems 193
6.16.2 Service Implementation 196
6.16.3 Factorization with the Quadratic Sieve 196
Contents xi
6.16.4 Service Architecture 197
6.16.5 Step 1: Obtaining the Factorization code 197
6.16.6 Step 2: Building a Factorization Service Interface 198
6.16.7 Step 3: Creating the Service Implementation 198
6.16.8 Step 4: Creating a Service Deployment Descriptor 200
6.16.9 Step 5: Creating the Schema Files and Grid Archives (GARs) 201
6.16.10 Step 6: Deploying the Service to the GT3 Container 210
6.16.11 Step 7: Writing a Service Client 211
6.16.12 Step 8: Running the Factorization Client 214
6.17 Summary 214
Part III The Globus Toolkit 217
7 Core 219
7.1 Concepts 219
7.2 Architecture 220
7.3 Hosting Environments 221
7.3.1 Apache Tomcat 222
7.3.2 WebSphere Application Server v5.1 222
7.3.3 Troubleshooting WebSphere 227
7.3.4 Other Hosting Environments 229
7.4 Summary 229
8 Security 231
8.1 Grid Security Infrastructure (GSI) Overview 232
8.2 Certificate Basics 232
8.2.1 Understanding X.509 Certificates 232
8.2.2 Typical Public Key Infrastructure (PKI) Model 233
8.2.3 Certificate Fields 234
8.2.4 GSI Proxies 235
8.2.5 Certificate Generation 236
8.2.6 Creating a Self-Signed User Certificate and Private Key 237
xii Contents
Contents xiii
8.2.7 Creating Credentials from User Certificates 238
8.2.8 Creating a Certificate Request 241
8.3 Java Certificate Services - Web Tools for X509 Certificates 247
8.3.1 Overview 247
8.3.2 Installing JCS on a Windows or Linux host 248
8.3.3 Deploying the Web-Based Tool on Tomcat 248
8.3.4 Accessing the Web-Based Tool 248
8.3.5 Creating a Certificate Request 248
8.3.6 Signing a Certificate Request 250
8.3.7 Creating a self-Signed (CA) Certificate and Private Key 250
8.3.8 Using the Command-Line - Based Tools 250
8.3.9 Creating a Certificate Request from the Command Line 251
8.3.10 Signing a Certificate Request from the Command Line 251
8.3.11 Getting Information on a X509 Certificate 251
8.3.12 Creating a Binary Distribution from Source 251
8.4 Globus Toolkit 3 Security Libraries 252
8.4.1 Creating Proxies from Default Certificates Within a
Web Application 252
8.4.2 Configuring Local Certificates for Testing 259
8.4.3 Creating Certificates and Private Keys 260
8.4.4 Troubleshooting: Security Provider Problems 267
8.5 Summary 268
9 Resource Management 269
9.1 Web Services GRAM Architecture (GT3) 270
9.1.1 Fault Tolerance 271
9.1.2 Resource Specification Language (RSL) 272
9.1.3 MMJFS Configuration 274
9.1.4 Troubleshooting 275
9.1.5 A Custom GRAM Client for MMJFS 278
9.1.6 Listening for Job Status Updates and Job Output 280
9.1.7 The GASS Server 282
9.1.8 RSL Submission 282
9.1.9 Security 287
9.1.10 MMJFS Performance 290
9.2 Pre Web Services GRAM Architecture (GT2) 291
9.2.1 Jobs in GT2 291
9.2.2 Job Submission Modes 291
9.2.3 Resource Specification Language (RSL) 292
9.2.4 Security Infrastructure 292
9.2.5 A Java Program for Job Submission on GT2 292
9.2.6 Listening for Job Output 294
9.2.7 Handling Received Output 295
9.2.8 Sending the GRAM Job Request 297
9.2.9 Job Submission Test 299
9.3 Summary 301
10 Data Management 303
10.1 GridFTP 304
10.1.1 Overview 304
10.1.2 Connecting to a GridFTP Server 305
10.1.3 Converting Legacy Proxies to GSS Credentials 307
10.1.4 Transferring Data 307
10.1.5 Transferring Multiple Files 309
10.1.6 Parallel Transfers 309
10.1.7 Testing the Transfer 311
10.1.8 Troubleshooting Tips 315
10.2 Transferring Files Using Multiple Protocols 316
10.3 Summary 319
11 Information Services 321
11.1 WS Information Services (MDS3) 322
11.1.1 Information Model 322
xiv Contents
Contents xv
11.1.2 Data Collection 322
11.1.3 Aggregation 322
11.1.4 Queries 323
11.1.5 User Interfaces 323
11.1.6 Security 324
11.2 Querying Default Data Providers 324
11.2.1 Enable the ServiceDataProvider and DataAggregation in
the Service Browser GUI 324
11.2.2 Test the default SystemInformation Service Data Provider
via the OGSA Service Browser GUI 325
11.3 Custom MDS3 Data Providers 326
11.3.1 GT 3.2 Core Data Providers 326
11.3.2 Provider Interfaces 327
11.3.3 The Simplest Case 328
11.3.4 The Real World: An MDS3 Information Provider for
Remote Schedulers Using SSH 330
11.4 Pre-WS Information Services (MDS2) 362
11.4.1 Architecture: GRIS and GIIS 362
11.4.2 Implementing a Grid Information Provider for MDS2 363
11.4.3 Custom MDS2 clients 373
11.5 Summary 379
12 Commodity Grid Kits (CoGS) 381
12.1 Overview 382
12.2 Language Bindings 382
12.3 Java 383
12.3.1 Installation Requirements and Configuration 383
12.3.2 Basic Java CoG Services 383
12.3.3 Security 384
12.3.4 Resource Management 393
12.3.5 Data Management 393
12.3.6 Data Transfer Troubleshooting 396
12.4 Other Language Bindings 397
12.4.1 Python 397
12.4.2 PERL 398
12.5 Summary 399
13 Web Services Resource Framework (WSRF) 401
13.1 Understanding WSRF 402
13.1.1 Stateless versus Stateful Services 402
13.1.2 WS-Resource 402
13.2 WSRF and OGSI 403
13.2.1 OGSI Evolution 403
13.2.2 From OGSI to WSRF 404
13.3 WSRF Normative Specifications 405
13.3.1 WS-ResourceProperties 406
13.3.2 WS-Addressing 408
13.3.3 WS-Resource Lifecycle 409
13.3.4 WS-ServiceGroup 410
13.3.5 WS-BaseFaults 410
13.3.6 WS-Notification 410
13.3.7 WS-Resource Security 411
13.4 GT4 and WSRF: Stateful Services for Grid Environments 411
13.4.1 Overview 411
13.4.2 Newcomers: WS-Components 411
13.4.3 Old Protocols, New Faces 412
13.5 Service Example: A WSRF Large Integer Factorization (LIF) Service 414
13.5.1 Web Service Description (WSDL) File 414
13.5.2 Resource Implementation 417
13.5.3 Service Implementation 421
13.5.4 Service Client 423
13.5.5 Enabling Security 428
13.5.6 Build and Deployment 428
xvi Contents
Contents xvii
13.5.7 Runtime Test 428
13.6 Summary 429
Part IV The Message Passing Interface (MPI) Standard 431
14 The Message Passing Interface (MPI) Standard 433
14.1 Overview 434
14.2 Procedures and Arguments 435
14.3 Data Types 435
14.3.1 Opaque Objects 435
14.3.2 Arrays and Constants 436
14.3.3 Other Data Types 436
14.4 Processes 436
14.5 Error Handling 437
14.6 Platform Independence 437
14.7 Point-to-Point Communication 437
14.7.1 Message Datatype Conversions 439
14.7.2 Message Envelopes 440
14.7.3 Data Type Matching and Conversion 441
14.7.4 Communication Modes 442
14.7.5 Buffering 442
14.7.6 Nonblocking Communication 443
14.7.7 Probe and Cancel 444
14.7.8 Derived Data Types 446
14.7.9 Pack/Unpack 447
14.8 Collective Communication 448
14.8.1 Barrier synchronization (MPI_BARRIER) 449
14.8.2 Broadcast (MPI_BCAST) 449
14.8.3 Gather (MPI_GATHER) 449
14.8.4 Scatter (MPI_SCATTER) 450
14.8.5 Gather-to-All (MPI_ALLGATHER) 451
14.8.6 All to All (MPI_ALLTOALL) 451
14.8.7 Global Reduction 451
14.8.8 Reduce Scatter (MPI_REDUCE_SCATTER) 454
14.8.9 Scan 454
14.8.10 Avoiding Deadlocks 454
14.8.11 Cyclic Dependencies 454
14.9 Groups, Contexts, and Communicators 455
14.9.1 Predefined Communicators 456
14.9.2 Miscellaneous Group and Communicator Examples 456
14.9.3 Intra-Communicators versus Inter-Communicators 458
14.10 Process Topologies 458
14.10.1 Overview 458
14.10.2 Virtual Topologies 459
14.11 Summary 459
15 Message Passing Interface (MPI) Standard 2.0 461
15.1 Overview 462
15.2 Changes Since Version 1 462
15.2.1 Deprecated Names and Functions 462
15.2.2 Processes 463
15.3 Miscellaneous Enhancements 463
15.3.1 The MPI-2 Process Model 463
15.3.2 Process Management and Resource Control 464
15.3.3 Memory Allocation 464
15.3.4 New Predefined Data Types 464
15.3.5 MPI-I/O 465
15.3.6 File Views 466
15.4 MPI Examples: Common Parallel Algorithms 467
15.4.1 Prime Generation - The Sieve of Eratosthenes 467
15.4.2 Sparse Matrix Multiplication 471
15.4.3 MPI-IO: Parallel Writes and Reads 480
15.5 Summary 486
xviii Contents
16 MPICH2 - A Portable Implementation of MPI 489
16.1 Overview 490
16.1.1 Language Bindings 490
16.2 Quick Start Installation Transcript 490
16.2.1 Software Prerequisites 490
16.2.2 Unpack the Tar File into a Temporary Directory 491
16.2.3 Create an Installation Directory (Default Is /usr/local/bin) 491
16.2.4 Configure MPICH2, Specifying the Installation Directory 491
16.2.5 Build and Install MPICH2 491
16.2.6 Add the bin Subdirectory of the Installation Directory to
Your Path 491
16.2.7 Test the MPICH2 Daemon Ring 491
16.2.8 Check the Sanity of your MPD Daemon 492
16.2.9 Create a MPD Ring and Test It 492
16.2.10 Run Simple Commands 493
16.2.11 Run the Sample MPI Programs 493
16.3 Large Integer Factorization Revisited: The Number
Field Sieve (NFS) 494
16.3.1 GNFS Algorithm 494
16.3.2 Public Domain GNFS - Sieving Over a Factor Base 496
16.3.3 Sieve Algorithm 496
16.3.4 LIF Execution Times 501
16.3.5 Parallel Sieve Implementation in MPI 502
16.3.6 Avoiding Interleaved Messages 515
16.3.7 Compilation and Runtime Data 519
16.4 MPICH2 - WS-GRAM Integration 519
16.4.1 Scheduler Interface for MPICH2 519
16.4.2 MMJFS Service Configuration 525
16.4.3 Packaging advice 528
16.4.4 Testing and Troubleshooting 529
16.5 MPICH2 - MDS3 Integration 532
Contents xix
16.5.1 An Information Service Script for MPICH2 532
16.5.2 Configuration and Testing 534
16.6 Summary 535
Appendix A Source Code 537
Program Name: SimpleCA.java 537
Creating a Certificate Request 537
Signing a Certificate Request 537
Getting Information on a X.509 Certificate 537
Program Name: CertGenerator.java 538
Program Name: CertSigner.java 538
Program Name: CertManaager.java 538
Program Name: Certrq.jsp 538
Program Name: self-sign.jsp 538
Program Name: userhost.jsp 538
Appendix B About the CD-ROM 539
Index 541
xx Contents
|
