Reference Number: xxxxx Intel Restricted Secret INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. Intel products are not intended for use in medical, life saving, life sustaining applications. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked “reserved” or “undefined.” Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The Itanium processor may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or by visiting Intel's Web Site. Intel, Itanium and Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Copyright © 1999-2004, Intel Corporation. All right reserved. *Other names and brands may be claimed as the property of others. Ref No xxxxx Intel Restricted Secret s 1 Introduction......................................................................................................................23 1.1 Preface................................................................................................................23 1.2 CSI Layers...........................................................................................................23 1.2.1 Physical Layer........................................................................................23 1.2.2 Link Layer...............................................................................................24 1.2.3 Routing Layer.........................................................................................24 1.2.4 Transport Layer......................................................................................25 1.2.5 Protocol Layer........................................................................................25 1.2.6 Communication Granularity Between Layers.........................................26 1.3 Notes...................................................................................................................26 1.4 Definition of Terms..............................................................................................26 2 Platform Scope.................................................................................................................29 2.1 Desktop/Mobile Systems.....................................................................................29 2.2 Dual Processor Systems.....................................................................................30 2.3 Quad-Socket and 8-Socket Systems ..................................................................31 2.4 Large Scale System Architectures......................................................................32 2.5 Profiles ................................................................................................................33 3 Physical Layer..................................................................................................................35 3.1 Physical Layer Overview.....................................................................................35 3.2 Physical Layer Features for Desktop/Mobile Systems - UP Profile.....................36 3.3 Physical Layer Features for Dual Processor Systems - DP Profile.....................37 3.4 Physical Layer Features for 4 and 8 Socket Systems - Small MP Profile...........37 3.5 Physical Layer Features for Large Scale Systems - Large MP Profile................38 3.6 Summary of Physical Layer Features .................................................................39 3.7 Physical Layer Reset...........................................................................................40 3.7.1 Link Power Up and Initialization Sequence............................................40 3.7.2 Link-Up Identifier....................................................................................42 3.7.3 Physical Layer Clocking.........................................................................42 3.7.4 Cold Reset..............................................................................................43 3.7.5 Inband Reset..........................................................................................43 3.7.6 Soft Reset...............................................................................................45 3.7.7 Two Stage Initialization ..........................................................................45 3.7.8 Automatic Test Equipment (ATE) Initialization Mode.............................47 3.8 Interface Between Physical Layer and Link Layer ..............................................49 3.9 Logical Sub-Block................................................................................................50 3.9.1 Supported Link Widths...........................................................................50 3.9.2 Link Training Basics...............................................................................62 3.9.3 Logical Sub-block Finite State Machine.................................................68 3.9.4 Optional Low Power Modes – An Overview...........................................83 3.9.5 Link Low Power Modes..........................................................................86 3.9.6 Physical Layer Determinism Requirements ...........................................98 3.9.7 Periodic Link Retraining .......................................................................100 3.9.8 Forwarded Clock Fail-Safe Mode – Small MP and Large MP Profiles.100 3.9.9 Link Self-Healing – Large MP Profiles..................................................101 3.9.10 Support for Hot Detect – Small MP and Large MP Profiles..................101 3.9.11 Lane Reversal......................................................................................101 Ref No xxxxx Intel Restricted Secret 4 3.10 Physical Layer Register Interface .....................................................................105 3.10.1 CSI Physical Layer Mandatory Registers.............................................106 3.10.2 Optional Registers................................................................................121 3.10.3 Electrical Parameter Registers (Examples Only).................................125 3.10.4 Testability Tool-box Registers (Examples Only) ..................................127 3.11 Electrical Sub-Block Specifications and Budgets..............................................130 3.12 Definition of Terms............................................................................................131 CSI Link Layer................................................................................................................135 4.1 Message Class..................................................................................................135 4.1.1 Required Base Message Classes........................................................137 4.2 Virtual Networks................................................................................................137 4.2.1 Base Virtual Network Requirements ....................................................138 4.3 Credit/Debit Flow Control..................................................................................138 4.4 Link Layer Buffer/Credit Management ..............................................................139 4.5 Support For Link Layer Reliable Transmission.................................................139 4.6 Packet Definition ...............................................................................................140 4.6.1 Packet Format......................................................................................140 4.6.2 Packet Fields........................................................................................159 4.6.3 Mapping of the Protocol Layer to the Link Layer..................................169 4.6.4 Width Reduction...................................................................................172 4.6.5 Organization of Packets on the Physical Layer....................................173 4.7 Link Layer Control Messages............................................................................173 4.7.1 Special Packet Format.........................................................................173 4.7.2 Null Ctrl Flit...........................................................................................174 4.7.3 Link Level Retry Ctrl Flit.......................................................................175 4.7.4 Power Management Ctrl Flit.................................................................175 4.7.5 System Management Ctrl Flit...............................................................176 4.7.6 Parameter Exchange Ctrl Flit...............................................................176 4.7.7 Sync Flit ...............................................................................................180 4.7.8 Error Indication.....................................................................................180 4.7.9 Debug...................................................................................................180 4.7.10 Idle Flit..................................................................................................187 4.8 Flit Interleave.....................................................................................................187 4.8.1 Command Insert...................................................................................188 4.8.2 Scheduled Data Interleave (SDI) .........................................................189 4.9 Transmission Error Handling.............................................................................189 4.9.1 Error Detection.....................................................................................189 4.9.2 Error Recovery.....................................................................................193 4.10 Link Layer Initialization......................................................................................200 4.11 Link Layer Required Registers..........................................................................203 4.11.1 CSILCP - CSI Link Capability Register ................................................203 4.11.2 CSILCL - CSI Link Control Register.....................................................204 4.11.3 CSILS - CSI Link Status Register.........................................................205 4.11.4 CSILP0 - CSI Link Parameter 0 Register.............................................206 4.11.5 CSILP1 - CSI Link Parameter 1 Register.............................................206 4.11.6 CSILP2 - CSI Link Parameter 2 Register.............................................206 4.11.7 CSILP3 - CSI Link Parameter 3 Register.............................................207 4.11.8 CSILP4 - CSI Link Parameter 4 Register.............................................207 4.12 Link Layer Rules and Requirements.................................................................207 4.13 Open Issues......................................................................................................207 Ref No xxxxx Intel Restricted Secret .................................................................................................................209 5.1 Introduction........................................................................................................209 5.2 Routing Rules....................................................................................................209 5.3 Routing Step......................................................................................................210 5.3.1 Router Table Simplifications.................................................................212 5.4 Routing Algorithm..............................................................................................213 5.5 Routing at Source and Destination Agents .......................................................213 5.6 Routing Broadcast of Snoops............................................................................213 5.7 Usage Models ...................................................................................................215 5.7.1 Flexible Interconnect Topologies..........................................................215 5.7.2 Flexible Partition Management.............................................................216 5.8 CSI Components’ Compatibility.........................................................................217 5.9 Configuration Space and Associated Registers................................................217 5.10 Routing Packets Before Routing Table Setup...................................................218 5.11 Routing Table Setup after System Reset/Bootup..............................................218 5.12 Route Table Setup after Partition Reset............................................................221 5.12.1 Single Partition.....................................................................................221 5.12.2 Partition with Route Through Components ..........................................221 5.13 Implementation Notes .......................................................................................221 5.14 Open Issues......................................................................................................221 6 CSI Protocol Overview...................................................................................................223 6.1 Protocol Messages............................................................................................223 6.2 Protocol Agents.................................................................................................223 6.3 Transaction IDs.................................................................................................224 6.4 Open Issues......................................................................................................225 7 Address Decode.............................................................................................................227 7.1 CSI Addressing Model.......................................................................................227 7.1.1 Types of Addresses..............................................................................227 7.1.2 Addressing Mechanism........................................................................227 7.1.3 Classification of Address Regions........................................................229 7.1.4 Relationship Between Memory Attribute, Region Attribute and CSI Transactions232 7.1.5 Assumptions and Requirements on System Address Map..................233 7.1.6 CSI Addressing Model..........................................................................234 7.1.7 Addressing Model in a Partitioned System...........................................235 7.2 Address Decoder...............................................................................................236 7.2.1 Generic Source Address Decoder........................................................237 7.2.2 Target Address Decoder at the Memory Agent....................................240 7.3 NodeID Assignment and Address Subdivision..................................................241 7.3.1 NodeID Assignment .............................................................................241 7.3.2 Caching Agent Address Subdivision....................................................241 7.3.3 Home Agent Address Subdivision........................................................242 7.4 Address Decode Configurations........................................................................242 7.5 Support for Advanced RAS Features................................................................243 8 CSI Cache Coherence Protocol.....................................................................................245 8.1 Protocol Architecture.........................................................................................245 8.1.1 Caching agent......................................................................................246 8.1.2 Home Agent .........................................................................................247 8.2 Protocol Semantics ...........................................................................................247 Ref No xxxxx Intel Restricted Secret 1 Coherent Protocol Messages...............................................................247 8.2.2 Protocol Dependencies........................................................................253 8.3 Caching Agent Interface....................................................................................256 8.3.1 Transaction Phases .............................................................................257 8.3.2 Coherence Domain ..............................................................................257 8.3.3 Cache States........................................................................................258 8.3.4 Peer Caching Agent Responses to an Incoming Snoop During the Null Phase258 8.3.5 Peer Caching Agent’s Response to a Conflicting Snoop During the Request and Writeback Phases260 8.3.6 Peer Caching Agent’s Response to a Conflicting Incoming Snoop During the AckCnflt Phase260 8.3.7 Responding to Cmp_Fwd* or Cmp to End the AckCnflt Phase............261 8.4 Source Broadcast Home Agent Algorithm ........................................................262 8.4.1 Home agent architected state ..............................................................262 8.4.2 Interpreting Protocol Flow diagrams.....................................................263 8.4.3 Protocol Flows Illuminated ...................................................................263 8.4.4 Protocol Invariants ...............................................................................272 8.4.5 Capturing Ordering...............................................................................274 8.4.6 Managing Conflict Lists........................................................................276 8.4.7 Summary of the home agent algorithm................................................281 8.5 Scaling CSI With an Out-of-Order Network.......................................................282 8.5.1 Directory Structure Requirements........................................................283 8.5.2 Home Agent Microarchitectural Constraints.........................................284 8.5.3 Simple Protocol Flows..........................................................................285 8.5.4 Home Agent Algorithm Overview.........................................................287 8.5.5 Using Coarse Sharing lists...................................................................289 8.5.6 Protocol English flows..........................................................................291 8.6 Application Notes..............................................................................................296 8.6.1 Global Observation ..............................................................................296 8.6.2 Flush Cache Operation ........................................................................296 8.6.3 Partial Write to Coherent Space...........................................................296 8.7 Coherence Protocol Open Issues .....................................................................298 8.7.1 Arbitrary AckCnflt’s...............................................................................298 Non-Coherent Protocol ..................................................................................................299 9.1 Transaction List.................................................................................................299 9.2 Protocol Layer Dependencies...........................................................................300 9.2.1 Requester Rules ..................................................................................300 9.2.2 Target Rules.........................................................................................302 9.3 Non-Coherent Memory Transactions................................................................303 9.3.1 Non-coherent Write Transaction Flow..................................................303 9.3.2 Non-Coherent Read Transaction Flow.................................................306 9.3.3 “Don’t Snoop” Transaction Flow...........................................................307 9.3.4 Length and Alignment Rules................................................................308 9.4 Peer-to-Peer Transactions................................................................................309 9.5 Legacy I/O Transactions ...................................................................................310 9.5.1 Legacy I/O Write Transaction Flow......................................................310 9.5.2 Legacy I/O Read Transaction Flow......................................................310 9.5.3 Addressing, Length and Alignment Rules............................................311 9.6 Configuration Transactions ...............................................................................311 9.6.1 Configuration Write Transaction Flow..................................................312 Ref No xxxxx Intel Restricted Secret 3 9.6.3 Addressing, Length and Alignment Rules............................................314 9.7 Secure Non-Coherent Transactions..................................................................314 9.8 Broadcast Non-Coherent Transactions.............................................................314 9.8.1 Broadcast Dependency Lists................................................................315 9.8.2 Broadcast Mechanism..........................................................................316 9.8.3 Broadcast Ordering..............................................................................316 9.8.4 Scaling to Large Systems.....................................................................316 9.9 Interrupts and Related Transactions.................................................................317 9.10 Non-coherent Messages...................................................................................317 9.10.1 Legacy Platform Interrupt Support .......................................................320 9.10.2 Power Management Support................................................................321 9.10.3 Synchronization Messages ..................................................................321 9.10.4 Virtual Legacy Wire (VLW) Transactions .............................................323 9.10.5 Special Cycle Transactions..................................................................326 9.10.6 Atomic Access (Lock)...........................................................................327 9.11 Non-Coherent Registers List.............................................................................332 10 Interrupt and Related Operations...................................................................................335 10.1 Overview ...........................................................................................................335 10.1.1 Interrupt Model for Itanium®-Based Systems.......................................336 10.1.2 Interrupt Model for IA-32 Processor Family-Based Systems ...............336 10.2 Interrupt Delivery...............................................................................................339 10.2.1 Interrupt Delivery Assumptions ............................................................342 10.2.2 Interrupt Redirection.............................................................................343 10.2.3 Interrupt Delivery for Itanium® Processor-Based Systems...................345 10.2.4 Interrupt Delivery for IA-32-Based Systems.........................................346 10.3 Level Sensitive Interrupt and End Of Interrupt..................................................349 10.4 Miscellaneous Interrupts and Events ................................................................350 10.4.1 8259A Support .....................................................................................350 10.4.2 INIT.......................................................................................................350 10.4.3 NMI.......................................................................................................350 10.4.4 SMI.......................................................................................................350 10.4.5 PMI.......................................................................................................350 10.4.6 PCI INTA - INTD and PME...................................................................351 10.5 Interrupt Related Configuration.........................................................................351 10.6 Reference Documents.......................................................................................351 11 Fault Handling................................................................................................................353 11.1 Definitions..........................................................................................................353 11.2 Error Classification............................................................................................353 11.3 Error Reporting..................................................................................................354 11.3.1 Error Reporting Mechanisms................................................................354 11.3.2 Error Reporting Priority.........................................................................358 11.4 Fault Diagnosis..................................................................................................358 11.4.1 Hierarchical Transaction Timeout.........................................................358 11.4.2 Error Logging Guidelines......................................................................361 11.5 Error Containment in Partitioned Systems........................................................361 11.5.1 Error Propagation in Partitioned Systems............................................361 11.5.2 Error Containment Through Packet Elimination...................................362 Ref No xxxxx Intel Restricted Secret ...................................................................................................367 12.1 Introduction .......................................................................................................367 12.2 CSI Reset Domains...........................................................................................367 12.2.1 CSI Physical Layer and Lower Link Layer Reset Domain....................368 12.2.2 CSI Upper Link Layer Reset Domains .................................................369 12.2.3 Routing Layer or Crossbar Reset Domain ...........................................370 12.3 Signals Involved in Reset..................................................................................372 12.3.1 PWRGOOD Signal...............................................................................372 12.3.2 RESET Signal ......................................................................................372 12.3.3 CLOCK Signals....................................................................................372 12.3.4 Other Configuration Signals.................................................................373 12.4 Initialization Timeline.........................................................................................374 12.5 Firmware Classification.....................................................................................375 12.5.1 Routing of Firmware Accesses.............................................................375 12.6 Link Initialization................................................................................................376 12.6.1 Link Initialization Options .....................................................................376 12.6.2 Exchange of System/Socket Level Parameters...................................377 12.7 System BSP Determination...............................................................................378 12.8 CSI Component Initialization Requirements .....................................................379 12.8.1 Support for Fabric Initialization.............................................................379 12.8.2 Programming of CSI Structures ..........................................................380 12.9 Support for On-Line Addition.............................................................................383 12.10 Support for Partition Reset................................................................................384 12.11 Hardware Requirements...................................................................................384 12.12 Configuration Space and Associated Registers................................................385 13 System Management Support........................................................................................387 13.1 Introduction .......................................................................................................387 13.2 Configuration Address Space ...........................................................................388 13.3 Configuration Access Mechanisms...................................................................388 13.3.1 CSI Configuration Agent ......................................................................389 13.3.2 JTAG and SMBus ................................................................................389 13.3.3 MMCFG and CF8/CFC.........................................................................390 13.4 Protected Firmware...........................................................................................390 13.4.1 Configuration Management Mode (CM Mode).....................................391 13.4.2 IA-32 Processor System Management Mode (SMM)...........................396 14 Dynamic Reconfiguration...............................................................................................399 14.1 Introduction .......................................................................................................399 14.2 Partitioning Models............................................................................................399 14.2.1 Hard physical partitioning (HPPAR).....................................................400 14.2.2 Firm physical partitioning (FPPAR)......................................................400 14.2.3 Logical or software partitioning (LPAR)................................................401 14.2.4 Virtual partitioning (VPAR) ...................................................................402 14.3 OL_* Support ....................................................................................................402 14.3.1 Implementation Dependent Quiescence/De-Quiescence ....................403 14.3.2 Flows....................................................................................................404 14.3.3 Assumptions/Requirements .................................................................407 14.3.4 Configuration Space and Associated Registers...................................408 14.3.5 Need for a Quiesce During OL_* Events..............................................409 14.4 Use of System Service Processor during OL_* Operations..............................409 Ref No xxxxx Intel Restricted Secret 1 14.5.1 Online Addition of a Processor Node (With or Without Other Agents).412 14.5.2 Online Addition of a Memory only Node...............................................414 14.5.3 Online Addition of an I/O Hub Node only .............................................415 14.6 On Line Deletion of a Node...............................................................................416 14.6.1 On Line Deletion of a Processor Node.................................................416 14.6.2 On Line Deletion of a Memory Node....................................................418 14.6.3 On Line Deletion of an I/O Hub Node...................................................419 14.7 Multi-Partition Management with Shared Interconnect......................................420 14.7.1 Restricted Option..................................................................................420 14.7.2 Restricted Option - Variant...................................................................421 14.7.3 Flexible Option .....................................................................................422 14.8 Support for Sub-Socket Partitioning..................................................................424 14.8.1 Sub-Socket Partitioning via Node ids...................................................424 14.8.2 Sub-Socket Partitioning via Firm Partition ID.......................................424 14.9 Memory RAS.....................................................................................................425 14.9.1 Memory Migration.................................................................................425 14.9.2 Memory Mirroring.................................................................................428 14.10 Hardware Requirements, Etc............................................................................431 14.11 Implementation Notes .......................................................................................432 14.12 Open Issues/Notes............................................................................................432 14.13 List of Acronyms Used ......................................................................................433 15 Power Management.......................................................................................................435 15.1 Link Power Management...................................................................................435 15.1.1 Link Power States ................................................................................435 15.1.2 L0s Link State.......................................................................................436 15.1.3 L1 Link State ........................................................................................438 15.1.4 L2 Link State ........................................................................................439 15.1.5 Link Width Modulation..........................................................................439 15.2 Platform Power Management............................................................................441 15.2.1 Platform Power States..........................................................................441 15.2.2 P, T, and C-State Coordination............................................................442 15.2.3 S-State Coordination............................................................................451 15.3 Power Management Related Messages ...........................................................454 15.3.1 Platform Power Management Messages .............................................454 15.3.2 Link Power Management Messages ....................................................455 16 Quality of Service and Isochronous Operations.............................................................459 16.1 Quality of Service (QoS)/Isochronous Platform Requirements.........................459 16.1.1 Legacy ISOC........................................................................................459 16.1.2 PCI-Express* ISOC..............................................................................459 16.1.3 Integrated Graphics ISOC services......................................................460 16.1.4 QoS extensions - compatible w/ PCI-Express......................................461 16.2 ISOC - Message Classes, and Traffic Classes.................................................461 16.2.1 Message Class definition .....................................................................461 16.2.2 Traffic Class definition..........................................................................461 16.2.3 Mapping ISOC transactions to ICS and IDS ........................................462 16.3 Link Layer Packet Fields and ISOC Support.....................................................462 16.4 Link Layer - QoS packet extensions..................................................................463 16.5 Usage Models of Isochronous Traffic in Current Platforms...............................464 16.6 ISOC/QoS Support Restrictions........................................................................465 Ref No xxxxx Intel Restricted Secret ..........................................................................................................................467 17.1 LaGrande Technology Background Information ...............................................467 17.2 Secure Launch In CSI Systems ........................................................................468 17.2.1 Simple CSI Systems ............................................................................468 17.2.2 Complex CSI Systems .........................................................................469 17.3 Link Initialization Parameters ............................................................................469 17.4 Interprocessor Communication: LT Link Layer Messages................................469 17.5 Processor-to-Chipset Communication: Protocol Layer Messages....................470 18 Design for Test and Debug............................................................................................473 18.1 Introduction .......................................................................................................473 18.2 Design For ATE-Based Testing and Debugging Through CSI .........................473 18.2.1 Tester Assumptions .............................................................................473 18.2.2 Basic Requirement: Determinism.........................................................474 18.2.3 Supporting the HVM Test Flow and Tester Fleet.................................476 18.2.4 Debug “Through” CSI – Debugging Processor or Chipset Via CSI Interface 477 18.2.5 Debug and Test of the Logic Associated with CSI...............................478 18.2.6 Desktop Processor Specific Requirements..........................................478 18.2.7 Debug of HVM Patterns .......................................................................478 18.2.8 Summary .............................................................................................479 18.3 Component and System DV/EV/AnV ...............................................................479 18.3.1 CSI Component and System DV/EV/AnV Requirements.....................480 18.3.2 Tx Characterization..............................................................................481 18.3.3 Rx Characterization..............................................................................481 18.3.4 Interconnect Characterization ..............................................................482 18.3.5 Link Characterization ...........................................................................482 18.3.6 CSI Link Debug for DV/EV/AnV ...........................................................483 18.4 CSI Phy Layer DFx Tools..................................................................................483 18.4.1 Introduction ..........................................................................................483 18.4.2 Definitions ...........................................................................................484 18.4.3 Reset Sequence...................................................................................485 18.4.4 CSI Loopback.......................................................................................485 18.4.5 Loopback Modes..................................................................................486 18.4.6 Local vs. Remote Loopback.................................................................488 18.4.7 Loopback Test Sequence ....................................................................489 18.4.8 Loopback Entry ....................................................................................489 18.4.9 Loopback Control Register...................................................................491 18.4.10Loopback Status Register....................................................................495 18.4.11 Loopback Exit.......................................................................................495 18.4.12CSI Determinism..................................................................................497 18.4.13Repeater Requirements.......................................................................500 18.4.14 CSI Eye Margining ...............................................................................501 18.4.15 Eye Width Adjust – Transmitter ...........................................................504 18.4.16 Eye Height Adjust – Receiver ..............................................................506 4.0.1 Eye Width Adjust – Receiver................................................................507 18.4.17Structural Tests....................................................................................508 18.5 Pin Leakage Testing - Transmitter and Receiver..............................................509 18.6 CSI Post-Si System Debug Requirements........................................................509 18.6.1 System Debug Requirements ..............................................................509 A Glossary ....................................................................................................................................... 515 Ref No xxxxx Intel Restricted Secret 5 A.2 List of Acronyms.............................................................................................................516 B CSI Profile Attributes ....................................................................................................................519 B.1 CSI Profile Attributes......................................................................................................519 Future Extensions - Transport Layer .............................................................................................525 C.1 Introduction .....................................................................................................................525 C.2 Reliable, End-to-End Transmission.................................................................................525 C.3 CSI Support for Reliable Transmission...........................................................................526 C.3.1 Routing..............................................................................................................527 C.3.2 Sequence Number .............................................................................................527 C.3.3 Transport Layer CSI Transactions ....................................................................528 C.3.4 Sender Node ID.................................................................................................528 C.3.5 No Time-Out Field............................................................................................529 C.4 Usage Models ..................................................................................................................529 C.5 CSI Components’ Responsibilities and Other Implementation Issues............................530 C.6 Notes, Comments for Later Revisions.............................................................................531 D Future Extensions - PTC.G............................................................................................................533 D.1 PurgeTC Special Transaction..........................................................................................533 D.1.1 Purge TC Messages and Resource Requirements.............................................533 D.1.2 Purge TC Transaction Flow ..............................................................................534 D.2 CSI Component Initialization Requirements...................................................................535 D.2.1 Programming of CSI Structures .......................................................................536 D.2.2 Online Addition of a Processor Node (With or Without Other Agents)...........536 D.2.3 On Line Deletion of a Processor Node .............................................................536 D.3 Open Issues/Notes ...........................................................................................................536 E Post Silicon Validation ..................................................................................................................537 E.1 Post-Si Validation for CSI...............................................................................................537 E.1.1 Summarized List of CSI Post-Si Validation Requirements ............................537 E.1.2 CSI Monitoring Events ...................................................................................538 E.1.3 Event Counters ................................................................................................540 E.1.4 Error Injection...................................................................................................541 E.1.5 Diagnostic Information Registers ...................................................................542 E.1.6 Programmable Configuration Overrides .........................................................543 E.1.7 Programmable Timer/Counter Values ............................................................543 E.1.8 Event Injection .................................................................................................543 E.1.9 CSI HUB-Based System Validation Concept...................................................544 E.2 Further Information (for Intel use only) .........................................................................550 E.3 DF Manufacturing Reference..........................................................................................551 E.4 Tester DV Further information........................................................................................551 F An Implementation Agnostic Model of CSI 2-Hop Source Broadcast Coherence .....................................................................................................................................553 F.1 Introduction .....................................................................................................................553 F.2 What CSI-IAM does and does not cover.........................................................................554 F.3 Components of CSI-IAM ................................................................................................554 F.4 Data Type Declarations ...................................................................................................554 F.5 The Initial State of the System ........................................................................................558 F.6 Invariants .........................................................................................................................559 Ref No xxxxx Intel Restricted Secret 1 F.8 Protocol Tables and Their Semantic Mappings.............................................................. 561 F.9 Utility Sub-Routines ....................................................................................................... 588 F.10 A C Reference Model Derived from CSI-IAM .............................................................. 595 F.10.1 Configuration parameters................................................................................. 595 F.10.2 Data Type Declarations.................................................................................... 596 F.10.3 API Functions................................................................................................... 597 G An Implementation Agnostic Model for CSI 3-Hop Home Broadcast Coherence..................................................................................................................... 601 G.1 Introduction .................................................................................................................... 601 G.2 What CSI-IAM Does and Does Not Cover .................................................................... 602 G.3 Components of CSI-IAM ............................................................................................... 602 G.3.1 IAM Component Details .................................................................................. 602 G.4 Data Type Declaration.................................................................................................... 604 G.5 The Initial State of the System ....................................................................................... 608 G.6 The Invariants ................................................................................................................. 611 G.7 Actions and Their Parameters......................................................................................... 612 G.8 Utility Sub-Routines ....................................................................................................... 662 Figures 1-1 Hierarchical Ordering of CSI Interface Layers ................................................................ 23 1-2 CSI Interface Layer Details (Routing and Transport Layers Not Shown) ....................... 24 1-3 CSI Interface Layer Details (Transport Layer Not Shown).............................................. 25 2-1 Schematic of an Intel® Itanium® Processor with CSI-Based Links Interface.................. 29 2-2 CSI-Based Uniprocessor Systems .................................................................................... 30 2-3 CSI-Based Dual Processor Systems ................................................................................. 30 2-4 4-Socket CSI-Based Platform........................................................................................... 31 2-5 4-Socket and 8-Socket CSI Systems ................................................................................ 31 2-6 Large Scale “Flat” Architecture........................................................................................ 32 2-7 Scalable Hierarchical System with OEM “Node Controllers”......................................... 33 3-1 CSI Layer Hierarchy......................................................................................................... 35 3-2 Physical Layer Power Up and Initialization Sequence – An Example............................. 41 3-3 Inband Reset Sequence Initiated by Port A to Port B....................................................... 44 3-4 Relationship between Phase Interpolator Training Pattern and Forwarded Clock Phase dur ing First Initialization Stage46 3-5 Interface Between Physical Layer and Link Layer – An Example .................................. 49 3-6 Mux Scheme for Link Width Support .............................................................................. 53 3-7 Physical Bit Swizzling...................................................................................................... 55 3-8 Sequence of Events for Acquiring Handshake Attributes ................................................ 64 3-9 State Transition Using Handshake Attributes .................................................................. 65 3-10 Logical Sub-block State Diagram..................................................................................... 68 3-11 Detect Sub-States.............................................................................................................. 70 3-12 Polling Sub-states ............................................................................................................. 74 3-13 Computing Lane-to-Lane Deskew – An Example ........................................................... 75 3-14 Config Sub-States ............................................................................................................. 79 3-15 Logical Sub-block State Diagram with Optional Low Power Modes .............................. 83 3-16 L0s Entry Sequence ..........................................................................................................87 3-17 L0s Exit Sequence ............................................................................................................ 89 3-18 Link Width Modulation Sequence.................................................................................... 93 Ref No xxxxx Intel Restricted Secret 7 3-20 Link Formed with a Straight Connection (No Lane Reversal Required)........................102 3-21 Daughter Card Topology - An Example .........................................................................102 3-22 Lane Reversal – An Example..........................................................................................103 3-23 Routing Guidelines for a Bifurcated Port using Lane Reversal on Both Halves ............104 3-24 Routing Guidelines for a Bifurcated Port Using Straight Connections on Both Halves.105 4-1 Special Packet Interleave Example .................................................................................188 4-2 Command Insert Interleave Example ..............................................................................189 4-3 Rolling CRC Scheme ......................................................................................................191 4-4 Error Detection on the Received flit Using Rolling CRC ...............................................191 4-5 Retry Queue and Related Pointers...................................................................................195 5-1 Routing Layer Functionality – 1......................................................................................211 5-2 Routing Layer Functionality – 2......................................................................................212 5-3 Abstract Structure of the Routing Table..........................................................................212 5-4 Illustrating Firmware Hub Connectivity Options............................................................219 5-5 Route Table Setup Using Breadth First Order ................................................................220 7-1 View of Types of Addresses in the System.....................................................................228 7-2 Itanium® Processor and IA-32 Addressing Models ........................................................234 7-3 Source Address Decoder at Requesting Agent................................................................237 7-4 Target Address Decoder at a Memory Agent..................................................................240 8-1 Protocol Architecture.......................................................................................................245 8-2 Caching Agent Architected State ....................................................................................246 8-3 A Visual Representation of Dependencies Within a Protocol Channel ..........................254 8-4 Home Agent Architected State........................................................................................262 8-5 Protocol Flow Legend .....................................................................................................263 8-6 Uncached RdData Request..............................................................................................264 8-7 Cached RdInvOwn Request ............................................................................................265 8-8 Standard Writeback Flow................................................................................................265 8-9 Generating a RspCnflt on a conflicting incoming Snoop................................................266 8-10 Sending an AckCnflt Due to a Conflicting Snoop ..........................................................267 8-11 Conflict Case Requiring FrcAckCnflt Flow....................................................................268 8-12 Conflict Case Continued from Figure 8-9 and Figure 8-10 ............................................269 8-13 WbMtoE Conflict ............................................................................................................270 8-14 WbMtoI Conflict .............................................................................................................271 8-15 Buried HITM Flow..........................................................................................................272 8-16 RspFwd Ordering Required.............................................................................................275 8-17 Writeback Ordering At the Home Agent.........................................................................276 8-18 Case Requiring a FrcAckCnflt to Resolve ......................................................................278 8-19 RdData Request Fetching an E-State Line and Setting Dir State....................................285 8-20 RdInvOwn Causing Invalidation of S-State Copies........................................................286 8-21 RdInvOwn Request HITM ..............................................................................................286 8-22 WbIData Arriving – We Discard Any WbMto* Message ..............................................287 8-23 Early Conflict Resolved by Detecting Request from Agent on Sharing List..................288 8-24 Late Conflict Resolved by Waiting for an AckCnflt.......................................................288 8-25 Buried HITM Case ..........................................................................................................289 8-26 Using the FrcAckCnflt/AckCnflt Handshake for a RdCode in Coarse Sharing .............290 8-27 Transiting from Explicit Sharers to Coarse Sharing........................................................291 8-28 Partial write to coherent space, Hit M.............................................................................297 8-29 Partial Write to Coherent Space, Conflict Case ..............................................................298 9-1 Non-Coherent Write Transaction Flow...........................................................................303 9-2 Non-Coherent Write Combinable Write Transaction Flow ............................................306 Ref No xxxxx Intel Restricted Secret 7 9-4 Legacy I/O Write Transaction Flow............................................................................... 310 9-5 Legacy I/O Read Transaction Flow................................................................................ 311 9-6 Configuration Write Transaction Flow........................................................................... 313 9-7 Configuration Read Transaction Flow ........................................................................... 313 9-8 Non-coherent Broadcast Example (IntPhysical) ............................................................ 316 9-9 Example Lock Flow........................................................................................................ 328 10-1 Interrupt Architecture Overview..................................................................................... 335 10-2 Address encoding in IntPhysical and IntLogical Requests ............................................ 340 10-3 Data field of IntPhysical and IntLogical Requests ......................................................... 340 10-4 Address field of IntPrioUpd Request.............................................................................. 344 10-5 Data field of IntPrioUpd Request ................................................................................... 345 10-6 Data field of NcMsgBEOI Request ................................................................................. 350 11-1 Illustration of Error Propagation Across Partitions ........................................................ 362 11-2 CSI Message Class Hierarchy ........................................................................................ 363 12-1 Reset Domains in CSI Components ............................................................................... 368 12-2 Example System Topology Diagram.............................................................................. 380 13-1 Logical View of Access Paths to CSI Configuration Registers ..................................... 387 13-2 Address Conversion Rules between Core & CSI Addresses (SMall MP) ..................... 393 13-3 Address Conversion Rules between Core & CSI Addresses (Large MP)...................... 394 13-4 Legacy SMM Memory Layout ....................................................................................... 396 13-5 IA-32 SMM Memory Layout in a CSI-Based System ................................................... 397 14-1 Hard Physical Partitioning Example............................................................................... 400 14-2 Firm Physical Partitioning Example............................................................................... 401 14-3 Logical Partitioning Example......................................................................................... 401 14-4 Virtual Partitioning Example.......................................................................................... 402 14-5 Illustrating Addition of a Node to a Running System .................................................... 411 14-6 Illustrating Removal of a Node from a Running System ............................................... 416 14-7 Multi-Partition Management - Restricted Option........................................................... 421 14-8 Multi-Partition Management- Restricted Option-Variant .............................................. 422 14-9 Multi-Partition Management - Flexible Option.............................................................. 423 14-10 Mirroring Support for Migration: Wt-Wt and Rd-Wt Mirroring ................................... 427 14-11 PMI/SMI Generation Sequence During OL_A Events .................................................. 432 15-1 Simple Lower Power State Example (Incomplete) ........................................................ 444 15-2 Lowering Power State Attempt With 1 Node Retry....................................................... 446 15-3 Lowering Power State With 2 Node Retr....................................................................... 447 15-4 Increasing from C4 to C0 State and Induced Retries on 2 Nodes .................................. 448 15-5 Conflict example - Request Passes Own Response........................................................ 450 15-6 S-State Entry Example.................................................................................................... 452 17-1 Transitive Trust Model ................................................................................................... 468 17-2 LT Link Layer Messages................................................................................................ 470 18-1 CSI Link Generic Diagram............................................................................................. 484 18-2 Segregated vs. Integrated Transceiver Floor Plans in Silicon ........................................ 486 18-3 Loopback Modes in CS .................................................................................................. 487 18-4 Local vs. Remote Loopback in CSI................................................................................ 488 18-5 Loopback Entry Timing Diagram................................................................................... 490 18-6 Loopback Entry Flow Diagram ...................................................................................... 491 18-7 Slave Agent – Receiver Input Common Mode Override ............................................... 492 18-8 Master Agent – Receiver Strobe Override...................................................................... 492 18-9 Slave Agent – Receiver Strobe Override........................................................................ 493 18-10 Master Agent – Transmitter Driver Current Override.................................................... 493 Ref No xxxxx Intel Restricted Secret Slave Agent – Transmitter Drive Current Override ........................................................494 18-12 A Basic And Minimal Pattern Buffer Architecture.........................................................495 18-13 Loopback Exit Timing Diagram......................................................................................496 18-14 Loopback Exit Flow Diagram .........................................................................................497 18-15 Example of Clock Synthesis............................................................................................498 18-16 System Level Determinism Using Counters ...................................................................499 18-17 CSI Flit Synchronization to the Tester ............................................................................500 18-18 Transmitter Eye Height Adjust Using the Equalizer.......................................................503 18-19 Transmitter Eye Height Adjust Using I-Comp Settings..................................................504 18-20 Transmitter Eye Width Adjust Using “Jitter Injection” ..................................................505 18-21 Transmitter Eye Width Adjust Using “Jitter Injection” Control Register.......................506 18-22 Receiver Eye Height Adjust Control Register.................................................................507 18-23 Receiver Eye Width Adjust by Overriding the PI Control Register ...............................508 C-1 Concept of Transport Layer Retry...................................................................................526 C-2 Interfacing of Components with and without Transport Layer.......................................530 D-1 Purge TC Transaction Flow.............................................................................................535 E-1 Histogram for FSB In-Order Queue................................................................................539 E-2 General Validation Structure...........................................................................................545 E-3 HVA Layered Architecture .............................................................................................546 E-4 HVA Data Link Level Traffic .........................................................................................547 E-5 HVA Data Link Layer Structure .....................................................................................548 E-6 HVA PHY Initialization Behavior ..................................................................................549 E-7 HVA in the Multi-linked System ....................................................................................549 E-8 HVA Implementation Structure ......................................................................................550 Tables 3-1 Physical Layer Features Supported in each CSI Profile....................................................39 3-2 Inband Reset Events for Figure 3-3...................................................................................44 3-3 ATE Initialization Mode - ATE Tx and DUT Rx .............................................................48 3-4 ATE Initialization Mode - ATE Rx and DUT Tx .............................................................48 3-5 Flit Format.........................................................................................................................50 3-6 Flit Format and Phit Order – Full Width Link ..................................................................51 3-7 Flit Format and Phit Order – Half Width Link..................................................................51 3-8 Flit Format and Phit Order – Quarter Width Link.............................................................51 3-9 Physical Pin Numbering and Clock Position on a Link with 20 Lanes.............................55 3-10 Link Map for Supported Link Widths...............................................................................56 3-11 Examples of Width Capability Indicator (WCI) ...............................................................57 3-12 CRC and Side-band Fields – Full Width Link ..................................................................58 3-13 CRC and Side-band Fields –- Half Width Link ................................................................58 3-14 CRC and Side-band Fields – Quarter Width Link.............................................................58 3-15 Pins Depopulated on Narrow Physical Interfaces .............................................................59 3-16 Narrow Physical Interface - Pin Map and Internal Representation...................................60 3-17 Summary of Narrow Physical Interfaces...........................................................................60 3-18 Physical Pin Numbering and Clock Position on a Link with 10 Lanes.............................61 3-19 Pin Map for Implementations Supporting Port Bifurcation ..............................................61 3-20 Training Sequence (TSx) Format ......................................................................................62 3-21 Summary of Handshake Attributes ...................................................................................63 3-22 Link Initialization Time Out Values..................................................................................67 3-23 Summary of "Disable/Start" state......................................................................................69 3-24 Summary of Detect.1 Sub-State ........................................................................................71 Ref No xxxxx Intel Restricted Secret 2 3-26 Summary of Detect.3 Sub-State ....................................................................................... 73 3-27 Summer of Polling.1 Sub-State ........................................................................................ 74 3-28 Description of TS2 Training Sequence ............................................................................ 75 3-29 Summary of Polling.2 Sub-State ...................................................................................... 76 3-30 Description of TS3 Training Sequence ............................................................................ 76 3-31 Summary of Polling.3 Sub-State ...................................................................................... 78 3-32 Description of TS4 Training Sequence ............................................................................ 79 3-33 Summary of “Config.1” State........................................................................................... 80 3-34 Summary of “Config.2” State........................................................................................... 81 3-35 Description of TS5 Training Sequence ............................................................................ 82 3-36 Summary of “L0” State .................................................................................................... 83 3-37 Summary of Extended L0 State with Low Power Support .............................................. 84 3-38 Summary of L0s State ...................................................................................................... 85 3-39 Summary of L1 State........................................................................................................ 86 3-40 L0s Entry Events and Timers ........................................................................................... 87 3-41 L0s Exit Events and Timers.............................................................................................. 90 3-42 Link Width Modulation Events and Timers ..................................................................... 94 3-43 L1 Entry and Exit Events/Timers ..................................................................................... 97 3-44 Register Attribute Definitions ........................................................................................ 106 3-45 CSIPHCPR0: Physical Layer Capability Register 0 ...................................................... 106 3-46 CSIPHCPR1: Physical Layer Capability Register 1 ...................................................... 107 3-47 CSIPHCTR: Physical Layer Control Register................................................................ 108 3-48 CSIPHTDC: Tx Data Lane Control Register ................................................................. 109 3-49 CSIPHTDS: Tx Data Lane Termination Detection Status Register............................... 110 3-50 CSIPHRDC: Rx Data Lane Control Register................................................................. 110 3-51 CSIPHRDS: Rx Data Lane RxReady Status Register.................................................... 111 3-52 CSIPHPIS: Physical Layer Initialization Status Register............................................... 111 3-53 CSIPHPPS: Physical Layer Previous Initialization Status Register............................... 113 3-54 State Tracker Encoding .................................................................................................. 115 3-55 CSIPHWCI: Width Capability Indicator (WCI) Register .............................................. 116 3-56 CSIPHLMS: Lane Map Status Register ......................................................................... 116 3-57 CSIPHPLS: Physical Layer Link Status Register .......................................................... 116 3-58 CSIPHITV0: Initialization Time-Out Value Register 0 ................................................. 117 3-59 CSIPHITV1: Initialization Time-Out Value Register 1 ................................................. 118 3-60 CSIPHITV2: Initialization Time-out Value Register 2.................................................. 118 3-61 CSIPHITV3: Initialization Time-Out Value Register 3 ................................................. 118 3-62 CSIPHITV4: Initialization Time-Out Value Register 4 ................................................. 119 3-63 CSIPHLDC: Link Determinism Control Register.......................................................... 119 3-64 CSIPHLDS: Link Determinism Status register .............................................................. 120 3-65 CSIPHPRT: Periodic Retraining Timer Register ........................................................... 120 3-66 CSIPHDDS: Link Determinism Drift Buffer Status Register ........................................ 121 3-67 CSIPHPMR0: Power Management Register 0............................................................... 121 3-68 CSIPHPMR1: Power Management Register 1............................................................... 122 3-69 CSIPHPMR2: Power management Register 2 ............................................................... 123 3-70 CSIPHPMR3: Power Management Register 3............................................................... 124 3-71 CSIPHPMR4: Power Management Register 4............................................................... 125 3-72 CSITCR: Termination Control Register......................................................................... 125 3-73 CSIETE: Equalization Tap Enable Register................................................................... 126 3-74 CSIECR0: Equalization Coefficient Register 0.............................................................. 126 3-75 CSIECR1: Equalization Coefficient Register 1.............................................................. 126 Ref No xxxxx Intel Restricted Secret 7 3-77 CSIRLR[0-19]: RX Lane Register n ...............................................................................127 3-78 CSILCR: Loopback Control Register .............................................................................127 3-79 CSILLMC: Loop-Back Lane Mask Control Register .....................................................128 3-80 CSILMRC: Loop-Back Master Receiver Control Register.............................................128 3-81 CSILMTC: Loop-Back Master Transmitter Control Register ........................................128 3-82 CSILSRC: Loop-Back Slave Receiver Control Register ................................................128 3-83 CSILSTC: Loop-Back Slave Transmitter Control Register............................................129 3-84 CSILPR0: Loop-Back Pattern Register 0........................................................................129 3-85 CSILPR1: Loop-Back Pattern Register 1........................................................................129 3-86 CSILPI: Loop-Back Pattern Invert Register....................................................................129 3-87 CSILSR: Loop Back Status Register...............................................................................129 3-88 CSILSP0: Loop-Back Status Pattern Register 0 .............................................................130 3-89 CSILSP1: Loop-Back Status Pattern Register 1 .............................................................130 3-90 CSILSLF: Loop-Back Status Lane Failure Register.......................................................130 3-91 Physical Layer Glossary..................................................................................................131 4-1 Message Classes, Abbreviations and Ordering Requirements........................................136 4-2 Standard Address, SA UP/DP .........................................................................................140 4-3 Standard Address, SA SMP.............................................................................................141 4-4 Standard Coherence Address, SCA UP/DP.....................................................................141 4-5 Standard Coherence Address, SCA SMP........................................................................141 4-6 Standard Coherence, SCC UP/DP...................................................................................142 4-7 Standard Coherence, SCC SMP ......................................................................................142 4-8 Standard Complete With Data, SCD UP/DP...................................................................142 4-9 Standard Complete With Data, SCD SMP......................................................................143 4-10 Extended Address, EA UP/DP ........................................................................................143 4-11 Extended Address, EA SMP............................................................................................144 4-12 Extended Address, EA LMP ...........................................................................................144 4-13 Extended Coherence Address, ECA LMP.......................................................................145 4-14 Extended Coherence No Address, ECC LMP.................................................................145 4-15 Extended Complete with Data LMP................................................................................146 4-16 Non-Coherent Message, NCM UP/DP............................................................................147 4-17 Non-Coherent Message, NCM SMP ...............................................................................148 4-18 Non-Coherent Message, NCM LMP...............................................................................149 4-19 3 Flit EIC format UP/DP .................................................................................................150 4-20 3 Flit EIC format SMP ....................................................................................................151 4-21 3 Flit EIC format LMP ....................................................................................................152 4-22 Standard Data Response, SDR UP/DP............................................................................152 4-23 Standard Data Response, SDR SMP ...............................................................................153 4-24 Standard Data Write, SDW UP/DP.................................................................................153 4-25 Standard Data Write, SDW SMP ....................................................................................153 4-26 Extended Data Response, EDR LMP..............................................................................154 4-27 Extended Data Write, EDW LMP...................................................................................154 4-28 Extended Byte Enable Data Write, EBDW UP/DP.........................................................155 4-29 Extended Byte Enable Data Write, EBDW SMP............................................................156 4-30 Extended Byte Enable Data Write, EBDW LMP............................................................157 4-31 Data Flit Format, DF .......................................................................................................157 4-32 Peer-to-Peer Tunnel SMP................................................................................................158 4-33 Peer-to-Peer Tunnel LMP................................................................................................159 4-34 Packet Length Encoding UP/DP/SMP ............................................................................160 4-35 Packet Length Encoding LMP.........................................................................................160 Ref No xxxxx Intel Restricted Secret 1 4-37 Message Class Encoding SMP/LMP.............................................................................. 162 4-38 Virtual Network Encoding.............................................................................................. 162 4-39 VC Credit Field Encoding UP/DP.................................................................................. 163 4-40 VC Credit Field Encoding SMP/LMP............................................................................ 164 4-41 Ack Field Encoding ........................................................................................................ 165 4-42 Scheduled Data Interleave Encoding.............................................................................. 166 4-43 Transfer Size Encoding .................................................................................................. 166 4-44 Special Cycle Encoding - 6b -PL ................................................................................... 167 4-45 Response Status - 2b -PL................................................................................................ 168 4-46 Response Data State - 4b - PL ........................................................................................ 168 4-47 Response Data State Encoding ....................................................................................... 168 4-48 Mapping of the Protocol Layer to the Link Layer UP/DP/SMP/LMP ........................... 169 4-49 Generic form for Special Packet, ISP............................................................................. 173 4-50 Opcode Encoding for Special Packet ............................................................................. 174 4-51 Null Ctrl Flit ................................................................................................................... 174 4-52 Link Level Retry Messages ............................................................................................ 175 4-53 Power Management Ctrl Flit .......................................................................................... 175 4-54 Power Management Link Messages ............................................................................... 176 4-55 Parameter Exchange Messages....................................................................................... 176 4-56 PE.Parameter0 ................................................................................................................ 177 4-57 PE.Parameter1 ................................................................................................................ 177 4-58 PE.Parameter2 ................................................................................................................ 178 4-59 PE.Parameter3 ................................................................................................................ 179 4-60 PE.Parameter4 ................................................................................................................ 180 4-61 Standard Debug Messages.............................................................................................. 181 4-62 Generic Debug Ctrl Flit .................................................................................................. 181 4-63 Inband Debug Event Ctrl Flit ......................................................................................... 182 4-64 Debug Relative Timing Exposure Ctrl Flit..................................................................... 185 4-65 Idle Special Packet, ISP.................................................................................................. 187 4-66 CRC Computation - Full Width...................................................................................... 192 4-67 CRC Computation - Half Width..................................................................................... 192 4-68 CRC Computation - Quarter Width................................................................................ 192 4-69 Control Messages and Their Effect on Sender and Receiver States............................... 196 4-70 Remote Retry State Transitions...................................................................................... 196 4-71 Local Retry State Transitions ......................................................................................... 198 4-72 Description of Send Controller....................................................................................... 199 4-73 Processing of Received Flit ............................................................................................ 200 4-74 Link Init and Parameter Exchange State Machine ......................................................... 201 4-75 CSILCP Format .............................................................................................................. 203 4-76 CSILCL .......................................................................................................................... 204 4-77 CSILS ............................................................................................................................. 205 4-78 CSILP0 ........................................................................................................................... 206 4-79 CSILP1 ........................................................................................................................... 206 4-80 CSILP2 ........................................................................................................................... 207 4-81 CSILP3 ........................................................................................................................... 207 4-82 CSILP4 ........................................................................................................................... 207 5-1 Combinations of Protocol Options ................................................................................. 215 5-2 Routing Layer Needs for Different Usage Models......................................................... 216 5-3 Interfacing CSI Components with Different VNs .......................................................... 217 5-4 CSI Control and Status Registers Needed by the Routing Layer ................................... 217 Ref No xxxxx Intel Restricted Secret 4 7-1 Characteristics of CSI Address Regions..........................................................................231 7-2 Allowed Attribute Combinations for Decode Register Entries.......................................232 8-1 Message Name Abbreviations.........................................................................................248 8-2 Message Field Explanations............................................................................................248 8-3 Snoop Channel Messages................................................................................................248 8-4 Home Channel Request Messages...................................................................................249 8-5 Home Channel Writeback Messages...............................................................................249 8-6 Home Channel Snoop Responses....................................................................................250 8-7 Home Channel AckCnflt Message ..................................................................................251 8-8 Response Channel Data Messages ..................................................................................252 8-9 Response Channel Grant Messages.................................................................................252 8-10 Response Channel Completions and Forces....................................................................253 8-11 Permitted Message Dependencies in CSI........................................................................255 8-12 Cache States.....................................................................................................................258 8-13 Required Cache State for Request Types ........................................................................258 8-14 A Peer Caching Agent’s Response to an Incoming Snoop .............................................259 8-15 Peer Caching Agent’s Response to a Conflicting Incoming Snoop During Request Phase, before DataC_*/GntE Response260 8-16 Cmp_Fwd* State Transitions ..........................................................................................261 8-17 Useful definitions ............................................................................................................272 8-18 Home Agent Responses, No Implicit Forward, Null Conflict List .................................279 8-19 Home Agent Responses, No Implicit Forward, Non-Null Conflict List.........................279 8-20 Cmp_Fwd* Types Sent to the Owner .............................................................................280 8-21 Example Directory Format..............................................................................................283 9-1 Non-Coherent Message Name Abbreviations.................................................................299 9-2 Non-Coherent Requests...................................................................................................299 9-3 Example Read Completion Formatting...........................................................................308 9-4 Peer-to-Peer Transactions................................................................................................309 9-5 Broadcast Non-Coherent Transactions............................................................................314 9-6 Target Agent Lists for Broadcast Transactions...............................................................315 9-7 Non-coherent Message Encodings (all use Message Header Type) ...............................317 9-8 NcMsg Parameter Encoding............................................................................................319 9-9 CmpD Parameter Encoding (uses SCC Header) .............................................................320 9-10 Legacy Pins Descriptions and CSI Handling ..................................................................323 9-11 Legacy Pin Signalling......................................................................................................324 9-12 VLW Value Field Bits (10:0) Definition.........................................................................325 9-13 VLW Value Change Bits (10:0) Definition.....................................................................326 9-14 IA-32 Special Cycles.......................................................................................................326 9-15 Lock Types ......................................................................................................................327 9-16 Non-Coherent Logical Register List ...............................................................................333 10-1 Setting of A[51:2] in IntPhysical Requests for Itanium® Processors..............................340 10-2 Setting of A[51:2] in IntPhysical and IntLogical Requests for IA-32 Processors ..........340 10-4 Setting of Data[31:0] in IntPhysical and IntLogical Requests for IA-32 Processors......341 10-3 Setting of Data[31:0] in IntPhysical Requests for Itanium® Processors.........................341 10-5 CSI Interrupt Modes........................................................................................................342 10-6 Setting of A[51:2] in IntPrioUpd Request for Itanium® Processors ...............................344 10-7 Setting of A[51:2] in IntPrioUpd Request for IA-32 Processors ....................................344 10-8 Interrupt Delivery in IA-32 Processor-Based Systems ...................................................348 11-1 Timeout Levels for CSI Requests with Source Broadcast ..............................................359 12-1 Justification for Reset Domain Separation......................................................................368 Ref No xxxxx Intel Restricted Secret 9 12-3 Features of CSI Upper Link Layer Reset Domain ......................................................... 369 12-4 Features of CSI Routing Layer or Crossbar Reset Domain............................................ 371 12-5 Features of CSI Protocol Agent Reset Domain .............................................................. 371 12-6 Node Identifier Options .................................................................................................. 373 12-7 System Type Values ....................................................................................................... 377 12-8 CSI Control and Status Registers Needed for Reset and Initialization .......................... 385 13-1 Division of Protected Resources for Isolation................................................................ 392 13-2 Sub-Regions of the Protected Region............................................................................. 392 13-3 Protected and PAL Mode Access Privileges .................................................................. 395 13-4 CSEG Operating Parameters .......................................................................................... 398 14-1 Control and Status Registers Needed for Quiesce/De-Quiesce...................................... 403 14-2 CSI Control and Status Registers Needed for Dynamic Reconfiguration Operations ... 408 15-1 Link State Overview....................................................................................................... 435 15-2 PMReq Data Field Mapping........................................................................................... 454 15-3 PMReq State_Type Field Encoding ............................................................................... 454 15-4 Power Management Transition Response Data Field Mapping ..................................... 455 15-5 CmpD State_Type Field Encoding for Power Management .......................................... 455 15-6 PM.LinkL0sConfig Data Field Mapping........................................................................ 456 15-7 PM.LinkWidthConfig Data Field Mapping.................................................................... 456 16-1 Isochronous Command and Data.................................................................................... 462 16-2 ISOC Request Attributes ................................................................................................ 462 16-3 Mapping of Traffic-class examples - to CSI Request Attributes.................................... 463 B-1 CSI Profile Attributes ..................................................................................................... 519 D-2 CSI Profile Attributes ..................................................................................................... 536 F-3 Actions of CSI-IAM ....................................................................................................... 561 F-4 Action CacheNewReqInt................................................................................................ 562 F-5 Action CacheNewReqExt............................................................................................... 564 F-6 Action CacheRecvData................................................................................................... 566 F-7 Action CacheRecvCmp .................................................................................................. 567 F-8 Action CacheRecvFwd ................................................................................................... 569 F-9 Action CacheSnpOrbMiss .............................................................................................. 572 F-10 Action CacheSnpOrbHit................................................................................................. 576 F-11 Action HomeRecvReq.................................................................................................... 577 F-12 Action HomeRecvRsp .................................................................................................... 579 F-13 Action HomeRecvAckCmp............................................................................................ 581 F-14 Action HomeRecvAckFwd............................................................................................. 582 F-15 Action HomeRecvWbData ............................................................................................. 583 F-16 Action HomeSendDataCmp ........................................................................................... 585 G-2 Action CacheNewReqInt................................................................................................ 614 G-3 Action CacheNewReqExt............................................................................................... 615 G-4 Action CacheRecvData................................................................................................... 617 G-5 Action CacheRecvCmp .................................................................................................. 619 G-6 Action CacheRecvFwd ................................................................................................... 620 G-7 Action CacheSnpOrbMiss ............................................................................................. 622 G-8 Action CacheSnpOrbHit................................................................................................. 625 G-9 Action HomeRecvReq.................................................................................................... 627 G-10 Action HomeRecvExplicitWbReq ................................................................................. 630 G-11 Action HomePRBtoSPTNoCDM ................................................................................... 633 G-12 Action HomePRBtoSPTCDM........................................................................................ 636 G-13 Action HomeRecvSnpRspNoCDM................................................................................ 637 Ref No xxxxx Intel Restricted Secret 1 G-15 Action HomeRecvWbSnpRsp.........................................................................................643 G-16 Action HomeRecvImplicitWbData .................................................................................648 G-17 Action HomeRecvRspCnfltNoCDM...............................................................................652 G-18 Action HomeRecvRspCnfltCDM....................................................................................656 G-19 Action HomeRecvAckCnflt ............................................................................................657 G-20 Action HomeSPTReadyToRespondNoCDM..................................................................659 G-21 Action HomeSPTReadyToRespondCDM.......................................................................661 Ref No xxxxx Intel Restricted Secret Information contained in this document is subject to change. Revision Number Description Date 0.0 • This is a first version of the CSI Specification for review purpose only. Do not use this version of specification for design purpose. It requires team review. • This version is showing the draft of Link Layer, Cache Coherence Protocol and Non Coherent and Interrupt transactions (along with the Introduction). March 2003 0.1 • Updated all the chapters. April 2003 0.3 • All chapter were updated. May/June 2003 0.5 • Major changes have been made to most of the chapters. The ones without changes are Introduction, Physical layer, Power Management, Fault Handling, and Security. August 2003 0.55 • The Physical layer, Power Management, Dynamic Reconfiguration chapters were updated in this revision. The Implementation Agnostic Model appendix has been removed from the document. It will be published separately. August 2003 0.7 • Added concept of the profiles to the document. Used conditional text to identify UP, DP, small MP (SMP), large MP (LMP), IA-32 and Itanium processor family profiles. • All chapters have been updated, UP Appendix has been removed, glossary and agnostic models have been added as appendices. October 2003 0.75 • All chapters have been updated. Protocol Overview chapter added. Post silicon validation appendix added. PTC.G appendix added. April 2004 § Ref No xxxxx Intel Restricted Secret 1.1 Preface This document is the specification of Intel’s CSI - a cache-coherent, link-based interconnect specification for processor, chipset, and I/O bridge components. CSI can be used in a wide variety of desktop, mobile, and server platforms spanning IA-32 and Intel® Itanium® architectures. CSI also provides support for high performance I/O transfer between I/O nodes. It allows connection to standard I/O buses such as PCI Express*, PCI-X, PCI (including peer-to-peer communication support), AGP, etc. through appropriate bridges. 1.2 CSI Layers The functionality of CSI is partitioned into five layers, one or more of which is optional for certain platform options. Each layer performs a well-defined set of non-overlapping functions. This layering results in a modular architecture that is easier to specify, implement, and validate. It also allows for easier future upgrades to the interface by allowing fairly independent optimizations at each layer. The layers, shown in Figure 1-1, from bottom to top are: Physical, Link, Routing, Transport, and Protocol. Figure 1-1. Hierarchical Ordering of CSI Interface Layers Protocol Layer Transport Layer Routing Layer Link Layer Physical Layer Optional L Optional LOptional La aay yye eers rsrs The transport and Routing layers, shown dotted in Figure 1-1, are optional and needed for certain platform options only. In desktop/mobile and dual processor systems, for example, the functionality of the Routing layer is embedded in the Link layer - hence, this layer is not separate in such systems. 1.2.1 Physical Layer The Physical layer is responsible for fast electrical transfer of information on the physical medium. The physical link is point to point between two Link layer CSI agents and uses a differential signaling scheme called Scalable Copper Interconnect Differential (SCID). Ref No xxxxx Intel Restricted Secret Introduction Introduction 1.2.2 Link Layer The Link layer abstracts the Physical layer from the upper layers and provides the following services: reliable data transfer and flow control between two directly connected CSI agents, virtualization of the physical channel into multiple virtual channels and message classes. The virtual channels can be viewed as multiple virtual networks for use by the Routing, Transport, and Protocol layers. The Protocol layer relies on the message class abstraction to map a protocol message into a message class and, hence, to one or more virtual channels. 1.2.3 Routing Layer This layer provides a flexible and distributed way to route CSI packets from a source to a destination. The routing is based on the destination. In some platform options (e.g., uniprocessor and dual processor systems), this layer may not be explicit but could be part of the Link layer; in such a case, this layer is optional. It relies on the virtual channel and message class abstraction provided by the Link Layer to specify one or more pairs to route the packet on. The mechanism for routing is defined through implementation specific routing tables. Such a definition allows a variety of usage models, which are described in the specification. Figure 1-2. CSI Interface Layer Details (Routing and Transport Layers Not Shown) Protocol Layer Phy Layer CSI Packets CSI Agent Protocol Engines CoherenceOrderingInterruptI/O Electrical Transfer Electrical Transfer . . . Buffered Flow Control CSI Agent Protocol Engines CoherenceOrderingInterruptI/O Electrical Transfer Electrical Transfer . . . Buffered Flow Control Link Layer Phit Flit = F *Phit Packet = P * Flit Ref No xxxxx Intel Restricted Secret Figure 1-3. CSI Interface Layer Details (Transport Layer Not Shown) Protocol Layer Phy Layer CSI Packets Link Layer Phy Layer CSI Packets CSI Agent Protocol Engines CoherenceOrderingInterruptI/O Electrical Transfer Electrical Transfer . . . Buffered Flow Control Routing Tables CSI Agent Protocol Engines CoherenceOrderingInterruptI/O Electrical Transfer Electrical Transfer . . . Buffered Flow Control Routing Tables Electrical Transfer Electrical Transfer . . . Buffered Flow Control Routing Tables Routing Layer Link Layer Phit Phit Flit = F *Phit Flit = F *Phit Routing Layer Packet = P * Flit Packet = P * Flit Packet = P * Flit 1.2.4 Transport Layer This layer provides support for end-to-end reliable transmission between two CSI agents that each have this layer’s capability. It relies on the services provided by the Routing layer below it, while in turn providing reliable transmission support to the Protocol layer above it. The Transport layer is optional and is provided for systems which desire a higher degree of reliability usually at the cost of perhaps lower performance and increased bandwidth utilization. In such systems, the Transport Layer functionality may be isolated to a few CSI components - in such a case, the sub-fields in the CSI packet related to this layer are defined in these components only. Since this layer is optional, it is possible to have a platform architecture with no CSI agent implementing this layer. Further, it does not follow the hierarchical layering of CSI from an implementation viewpoint (See Appendix C, “Future Extensions - Transport Layer”). In the rest of this specification, the Transport Layer is not shown or assumed, unless explicitly mentioned. 1.2.5 Protocol Layer This layer implements the higher level communication protocol between nodes such as cache coherence (reads, writes, invalidations), ordering, peer-to-peer I/O, interrupt delivery, etc. CSI provides a flexible protocol which can scale from small to large systems. The write-invalidate protocol implements the MESIF states, where the MESI states have the usual connotation (Modified, Exclusive, Shared, Invalid), and the F state indicates a read-only forwarding state. The CSI protocol allows for source snooping (the requester initiates snoops of the caching agents), home snooping (home initiates snoops of the caching agents), or a combination of the two. It is permissible for the F state to be not used (for example, in home snooping based systems). The exact functionality of this layer depends on the platform architecture. The Protocol layer is bypassed in pure routing agents resulting in low latency transfer from sender to the receiver through the interconnection network (please see Figure 1-2). Ref No xxxxx Intel Restricted Secret Introduction Introduction 1.2.6 Communication Granularity Between Layers The data transfer unit at the Physical layer is called a phit (physical unit). The Link layer between two CSI agents communicate at a higher granularity called flit (flow control unit). A flit is the smallest granularity for flow control. A flit is made of multiple phits. The protocol, transport, and Routing layers communicate at the granularity of a packet. Each packet consists of one to many flits, depending on the packet type and the system configuration - thus, it may consist of one or more header flits optionally followed by a data payload consisting of multiple flits (please see Figure 1-2). In the rest of the specification, a CSI agent always refers to a protocol agent, unless explicitly mentioned otherwise. 1.3 Notes The conditional text tags have been used in the document to distinguish between various system profiles. System profiles are defined in following chapter. System profiles are marked with conditional text and specific colors. The following is the list describing conditional text used to describe profiles: • Sample of the text for UP description • Sample of the text for DP description • Sample of the text for SMP description • Sample of the text for LMP description • Sample of the text for IA-32 description • Sample of the text for Itanium processor family description • Sample of text using multiple conditional tags (ex. UP, DP) 1.4 Definition of Terms The terms defined in this section are frequently used in subsequent chapters of the specification. Additional terms are defined in the following chapters of the document to better describe the content of the material. The definition of terms will be consolidated in the future revision of the CSI specification. The complete list of definitions is provided in Appendix A, “Glossary.” Device Address Caching Agent Configuration Agent This is the address generated by the target node of an CSI transaction to access the physical memory or device location. This address is used on the I/O buses or on the memory interface. This address may be same as the physical address part of the system address or it may be translated through some (optional) mapping mechanism at the target. A protocol agent type which can perform reads & writes into coherent memory space. The logical owner of all platform configuration registers on a CSI agent or component. A component may define a separate CA for each CSI agent on the die or it may define a single CA to represent all the CSI agents on the die. In the latter case, configuration transactions destined Ref No xxxxx Intel Restricted Secret to CSRs in other CSI agents are logically targeted to the CA, which in turn completes the access within the die via implementation-specific mechanisms. Firmware agent A CSI agent capable of supplying boot firmware to processor cores. Home Agent A protocol agent type which is responsible for guarding access to a piece of coherent memory. I/O Agent A protocol agent type which is responsible for non-CSI I/O devices behind it. As a CSI initiator, an I/O Agent makes CSI requests on behalf of I/O devices and returns responses back to the I/O device. As a target, an I/O Agent is responsible for translating CSI requests to the protocol native to its I/O interface and returns I/O responses back to the CSI requester. Physical Address This is the operating system’s view of the address space in a partition. This is obtained by translating virtual address through the operating system page translation mechanism. This is also the address used by the cache coherency mechanism, which puts certain requirements on the mapping of coherent shared address space within and across partitions. Processor Agent The CSI interface to a logical processor. (this definition needs to be revised and will need to change as we better understand how interrupts, VLWs, etc. are partitioned in designs). Routing Agent A CSI agent which implements a routing step, routing a CSI packet from the input port of a router to the destination port based on the destination node id contained in the packet. A packet is routed from its source to its destination through a series of routing steps. System Address The system address is represented by the physical address and the target (home) node identifier, which points to a unique device address in a system. The addressing model allows same physical address from different source agents to map to different system address (e.g., private firmware space per processor agent) or to the same system address (e.g., shared memory space in a partition or across partitions) irrespective of partition boundaries. System address also includes the scope of hardware cache coherency. For example, a system may have identical physical memory addresses in different partitions, but with different home nodes and different scope of coherency and therefore distinct system addresses. Also note that in the source broadcast based cache coherency scheme, the home node identifier does not play a role in specifying the scope of coherency. Virtual Address This is the address used by the applications, device drivers and devices (if I/O agents support paging). § Ref No xxxxx Intel Restricted Secret Introduction Introduction Ref No xxxxx Intel Restricted Secret This chapter outlines the flexible platform architectural options that are possible with CSI-based interconnect. CSI can be used in a wide variety of desktop, mobile, and server platforms spanning IA-32 and Itanium architectures. Figure 2-1. Schematic of an Intel® Itanium® Processor with CSI-Based Links Interface Xbar Router/ Non-routing global links interface CSI Links P P P MemoryController(Optional) Processor Cores with split/ shared caches Memory Intf (Optional) Figure 2-1 shows a schematic of a processor with external CSI-based link interfaces. The processor may have 1 or more cores. In case multiple core are present they may share caches or have separate caches The processor may also support optional integrated memory controller(s). In addition, based on level of scalability support in the processor, it may include an integrated crossbar router and 1 or more external CSI link interface. In the rest of the chapter, we discuss the various system profiles that may be supported by different processor implementations. 2.1 Desktop/Mobile Systems Figure 2-2 shows two example configurations, each with a single socket. In each case the processor is directly connected to the chipset through a single CSI link. In the first configuration, the processor's main memory is supported through a memory controller on the chipset (that also has graphics related functionality). In the second configuration, the processor's memory is directly connected to the processor socket and the processor is assumed to have an integrated memory controller on die; the chipset primarily supports graphics related functionality. Both configurations have I/O connectivity and firmware through other chipsets with connectivity as shown in the Figure 2-2. Ref No xxxxx Intel Restricted Secret Platform Scope Platform Scope Figure 2-2. CSI-Based Uniprocessor Systems IA Processor Graphics + Memory Ctrl ICH Firmware Hub CSI Link LPC Bus DMI Memory PCI Express Links IA Processor + iMC Graphics + Memory Ctrl ICH Firmware Hub CSI Link LPC Bus DMI Graphics Memory Memory PCI Express Links To keep the focus primarily on the CSI-related parts of platform configurations, most other platform components are not shown in later sections. 2.2 Dual Processor Systems Figure 2-3. CSI-Based Dual Processor Systems PCI Express Links IA Processor Graphics + Memory Ctrl CSI Link Memory IA Processor IA Processor + iMC IO Hub CSI Link DMI IA Processor + iMC Memory PCI Express Links Optional Optional The dual processor options shown in Figure 2-3 represent two of several possible platform options. The first option has a centralized main-memory connected to the graphics controller. Each processor sockets has two CSI links, one connecting it to the graphics and memory controller and the other to the second processor socket. The second option assumes a distributed memory platform with each processor having part of the main memory directly connected to it. This option shows an I/O Hub connected to the processor sockets instead of the graphics controller and represents a yet another possible variation amongst dual processor platforms. In both the configurations shown, the optional direct processor to processor link helps provide additional network bandwidth as well as reduces latency to snoop caches on the other processor and that of direct cache-to-cache transfers of instructions/data. In each of the single processor (desktop) and dual processor platform configuration the processors need not support any special routing capability. Ref No xxxxx Intel Restricted Secret 2.3 Quad-Socket and 8-Socket Systems Figure 2-4 shows a 4-socket platform configuration where each processor socket is connected to every other processor socket (“fully-connected”) and also to a single I/O Hub through CSI links. This platform also has fully-distributed memory architecture. This architecture has high performance because of its rich interconnectivity which permits quick snoop resolution, fast memory and cache-to-cache transfers. Variants of this architecture could include the use of multiple I/O Hubs. A different version of the 4-socket platform (not shown here) could be a cost optimized one with a “square” interconnect between processors such that it requires 1 fewer CSI link per processor socket. Once again, multiple I/O Hub based solutions are also possible in this configuration. Figure 2-4. 4-Socket CSI-Based Platform IA Processor + iMC IA Processor + iMC Full or Half- width CSI Link IO Hub DMI Memory CSI Link IA Processor + iMC IA Processor + iMC PCI Express Links In Figure 2-5, abstract CSI-based platform topologies for 4-socket and 8-socket platforms are shown. Figure 2-5. 4-Socket and 8-Socket CSI Systems 0 1 2 3 IA Processor w/disabled iMC XMC IA Processor + iMC OR 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 In particular, the 4-socket abstraction represents the inter-processor interconnect for the platform shown in Figure 2-5 ignoring the interconnects to I/O and other platform components. The 8socket topology of Figure 2-5 shows cube-topology which utilizes 3 CSI links for inter-socket communication. Additional connectivity between processor sockets is possible with more CSI Ref No xxxxx Intel Restricted Secret Platform Scope Platform Scope links supported by a given Itanium processor implementation, as shown by the option on the right. This would improve the overall bandwidth available in the platform as well as improve its latency characteristics. It is feasible that 8-socket systems could be built with 4 or 5 inter-processor CSI links supported by a particular Itanium processor implementation. Such richer connectivity leads to higher performance and better RAS features. 2.4 Large Scale System Architectures Platform architectures that scale beyond 4 or 8 Itanium architecture-based processors could be enabled through OEM chipsets. Figure 2-6. Large Scale “Flat” Architecture CSI Link IA Processor + iMC IA Processor + iMC CSI Interconnection Network IO Hub XMCXMC PCI Express Links One approach to build general large-scale CSI-based platform example is shown in Figure 2-6. Here each processor interfaces to two (or more) external memory controller (XMC) chips through CSI links. The XMC is an OEM component which utilizes a scaled version of CSI with directory support and other features to enhance performance (e.g., directory cache) and RAS. The interconnect network topology can be general, especially, with additional OEM components such as routers and cable drivers. As opposed to the “flat” architecture of Figure 2-6, another way to build large scale systems is to use a hierarchical approach - the basic building block comprises of a n-socket node (where n is some small number); such nodes are, in turn, interconnected through OEM node controllers. The building block uses CSI interfaces while the node controllers could use CSI links or the OEM’s proprietary interfaces. Figure 2-7 shows an example of a node-controller based 4-socket platform architecture. Such an OEM designed node-controller could optionally support a remote memory cache, a partial or full directory, and a directory-cache for scalability, performance, and RAS reasons. Itanium processors which are used in large scale systems will have an internal router to support through routing. Ref No xxxxx Intel Restricted Secret Figure 2-7. Scalable Hierarchical System with OEM “Node Controllers” IA Processor + iMC IA Processor + iMC Memory CSI Link IA Processor + iMC IA Processor + iMC Node Controller Interconnection Network CSI-based or Proprietary Interconnect for Scale-up 2.5 Profiles A central notion in CSI is that of a profile. Since CSI targets a range of architectural platforms, the interface definition identifies the essential features that are common across this range and also features that are specific to a particular platform or set of platforms. These specific features form the profile. For example, the CSI features of a desktop profile would optimize cost and performance while those for a large system profile would target scalability and RASUM (reliability, availability, serviceability, usability, manageability). CSI has been carefully designed to permit both unification across profiles and optimizations targeted to each profile. The CSI profiles fall roughly in line with the architectural options introduced in this chapter: uniprocessor system that include both the desktops and mobile platforms, dual processor systems, small scale system (4 - 8 socket systems), and large scale system that typically more than 8 sockets. It has to be noted, however, that profile dependent fields permit certain features and, correspondingly, restrict other features - processor and chipset implementations targeting specific platforms utilize the needed combination of the profile dependent fields - hence there is not a strict mapping of profile dependent fields into exact architectural options. At the highest level, CSI packet headers have the standard format (1 flit) and the extended header format (2 flit). The standard format is expected to be used for all profiles except the large scale systems. The extended format permits, for example, larger addressability, higher number of CSI agents that can be supported in the system at the expense of additional interconnect bandwidth. The